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The  purpose  of  this  guidebook  is  to  provide  state-of-the-art  information 
about  the  selection  and  use  of  existing  software  reliability  models. 

Towards  this  objective,  we  have  presented  a  brief  summary  of  the  available 
models  backed  by  a  detailed  discussion  of  most  of  the  models  in  the 
appendices. 

One  of  the  difficulties  in  choosing  a  model  is  to  find  a  match  between  —— 
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the  testing  environment  and  a  class  of  models.  To  help  a  user  in  this 
process,  we  have  presented  a  detailed  discussion  of  most  of  the 
assumptions  that  characterise  the  various  software  reliability  models. 

The  process  of  developing  a  model  has  been  explained  in  detail  and 
illustrated  via  numerical  examples. 
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1.  INTRODUCTION 


1 . I  Background 

An  important  quality  attribute  of  a  computer  system 
is  the  degree  to  which  it  can  be  relied  upon  to  perform 
its  intended  function.  Evaluation,  prediction,  and  improve¬ 
ment  of  this  attribute  have  been  of  concern  to  designers  and 
users  of  computers  from  the  early  days  of  their  evolution. 
Until  the  late  sixties,  attention  was  almost  solely  on  the 
hardware  related  performance  of  the  system.  In  the  early 
seventies,  software  also  became  a  matter  of  concern,  pri¬ 
marily  due  to  a  continuing  increase  in  the  cost  of  software 
relative  to  hardware,  in  both  the  development  and  opera¬ 
tional  phases  of  the  system. 

Software  is  essentially  an  instrument  for  transforming 
a  discrete  set  of  inputs  into  a  discrete  set  of  outputs. 
Since,  to  a  large  extent,  software  is  produced  by  humans, 
the  finished  product  is  often  imperfect.  It  is  imperfect 
in  the  sense  that  a  discrepancy  exists  between  what  the 
program  can  do  and  what  the  user  or  the  computing  environ¬ 
ment  wants  it  to  do.  These  discrepancies  are  called  soft¬ 
ware  faults. 

Even  if  we  know  that  software  contains  faults,  we 
generally  do  not  know  their  exact  identity.  Currently, 
there  are  two  approaches  for  exposing  software  faults: 
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program  proving  and  program  testing.  Program  proving, 
though  formal  and  mathematical,  is  still  an  imperfect 
tool  for  verifying  program  correctness.  Program  testing 
is  more  practical  but  somewhat  heuristic. 

Due  to  the  imperfectness  of  these  approaches  in 
assuring  a  correct  program,  a  metric  is  needed  which  re¬ 
flects  the  degree  of  program  correctness  and  which  can  be 
used  in  planning  and  controlling  additional  resources  (time 
and  money)  needed  for  enhancing  softw  j  quality.  One  such 
quantifiable  metric  of  quality  that  h  -  become  popular  in 
software  engineering  practice  is  sof  .  reliability. 

A  number  of  models  have  been  proposed  during  the  last 
ten  years  for  assessing  software  reliability.  However,  very 

few  efforts  [ANG80 ,  GOE83 J  have  been  undertaken  to  evaluate 

* 

their  assumptions  and  limitations.  Also,  information 
about  the  applicability  of  these  models  during  various 
phases  of  software  development  has  been  lacking. 

This  report  presents  a  summary  and  evaluation  of  most 
of  the  available  models  for  software  reliability  assessment. 
A  discussion  of  software  quality,  software  testing,  and 
software  reliability  is  provided  in  Section  2,  and 
a  brief  description  of  the  times  between  failures  and 


* 

A  study  sponsored  by  RADC  is  currently  in  progress  to  assess 
the  applicability  of  selected  models  to  field  data  from  an 
on-going  project. 
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failure  count  software  reliability  models  is  given  in 
Sections  3  and  4,  respectively.  Fault  seeding  and  input 
domain  based  models  are  described  in  Section  5.  Assump¬ 
tions,  limitations  and  applicability  of  these  models  are 
discussed  in  Section  6.  A  step  by  step  procedure  for 
developing  a  software  reliability  model  is  given  in 
Section  7,  and  some  of  the  steps  are  illustrated  via 
numerical  examples. 


. 2  Limitations  and  Applicability  of  the  Guidebook 

The  purpose  of  this  guidebook  is  to  provide  a  compre¬ 
hensive  treatment  of  most  of  the  analytical  models  proposed 
during  the  last  ten  years  for  software  reliability  assess¬ 
ment.  As  stated  above,  the  guidebook  also  contains  a  dis¬ 
cussion  of  the  model  assumptions,  their  limitations  and 
their  applicability  during  various  phases  of  the  software 
life  cycle. 

The  state-of - the-art  of  software  reliability  modeling 
is  akin  to  a  moving  target.  It  was  felt,  however,  that  the 
time  had  come  to  document  the  relevant  material  about  the 
available  techniques  so  that  future  efforts  could  be  focussed 
on  resolving  the  unsolved  problems.  This  viewpoint  played 
a  key  role  in  the  treatment  of  the  material  presented  here. 
Specifically,  the  inclusion  of  any  model  in  the  guidebook 
does  not  necessarily  imply  that  such  a  model  is  the  right 
one  to  use  for  software  reliability  assessment.  A  decision 
about  the  usability  or  otherwise  of  a  model  in  a  given 
testing  situation  should  be  based  on  a  clear  understanding 
of  the  assumptic-ns  and  limitations  of  that  model  vis-a-vis 
the  actual  testing  process. 

The  following  points  should  be  helpful  in  determining 
the  limitations  and  applicability  of  this  guidebook. 

1.  The  guidebook  does  not  recommend  any  particular  model 
or  a  class  of  models.  It  encourages  the  user  to  under¬ 
stand  the  model  assumptions  and  its  limitations  before 


using  it  in  a  given  situation. 


2.  The  method-  " ngy  described  in  the  guidebook  for  developing 
a  reliability  model  is  a  general  one  and  should  be  usable 
in  many  situations.  However,  other  approaches  may  be 
mere  appropriate  in  certain  developing  .t  environments  and 
should  be  preferred. 

3.  Most  of  the  models  use  time  (execution  or  calendar)  as 
a  basis  for  studying  fault  occurrence  processes.  The 
entity  time  should  be  interpreted  in  a  broader  sense. 

Any  other  measure  which  may  be  more  relevant  in  a  given 
situation  can  be  used  in  lieu  of  time  if  the  model 
assumptions  are  satisfied.  Some  examples  of  such 
measures  are  lines  of  code  tested,  number  of  functions 
tested,  and  number  of  test  cases  executed. 

4.  The  models  described  in  the  guidebook  treat  the  soft¬ 
ware  product  as  a  black  box  at  a  macro  level.  They  do 
not  explicitly  take  into  account  the  effects  of  develop¬ 
ment  methodologies  and  support  tools.  If  such  con¬ 
siderations  are  of  interest  and  concern,  the  models 
described  here  may  not  be  the  appropriate  ones  to  use. 

5.  Most  of  the  models  apparently  were  developed  for  use 
during  the  system  testing  stage  to  estimate  soft¬ 
ware  reliability  in  the  field.  Many  of  these,  however, 
can  also  be  used  in  other  development  phases  if  the 
underlying  assumptions  are  satisfied. 
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2.  SOFTWARE  QUALITY,  TESTING,  AND  RELIABILITY 

2 . 1  Software  Quality  Problem 

The  importance  of  software  quality  in  determining  the 
performance  of  a  computer  system  has  been  well  recognized 
during  the  last  decade.  Currently,  the  low  quality  of 
software  is  the  limiting  factor  in  achieving  overall  system 
quality.  Among  the  reasons  for  low  software  quality  are 
the  need  for  extensive  human  involvement  in  software  develop¬ 
ment  and  the  uncertainty  and  complexity  of  system  applica¬ 
tions  . 

Software  quality  assessment  is  different  from  hard¬ 
ware  quality  assessment.  The  latter  is  primarily  concerned 
with  the  accuracy  of  fabricating  (copying)  the  hardware 
design.  Controlling  the  quality  of  fabricated  designs  is 
known  in  engineering  as  Quality  Control.  This  aspect  of 
quality  has  been  traditionally  quantified  by  using  statistical 
techniques.  The  quality  of  software,  on  the  other  hand,  is 
determined  primarily  by  the  quality  of  the  design.  The 
development  and  implementation  processes  that  go  into  the 
production  of  software  cannot  be  easily  quantified  at 
present . 

The  software  development  process  can  be  divided 
roughly  into  five  phases: 

(1)  Requirements  analysis.  Requirements  analysis 
involves  understanding  the  user  requirements 
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and  expressing  these  requirements  in  the  form 
of  formal  or  informal  specifications. 

(2)  Design.  The  design  phase  further  refines  the 
specifications  to  come  up  with  computer  com¬ 
patible  specifications. 

(3)  Programming.  Programming  is  the  act  of  imple¬ 
menting  the  specifications  in  a  specified 
computing  environment.  In  general,  programming 
involves  coding;  however,  pre-packaged  modules 
or  program  generations  may  be  available  to 
satisfy  the  specifications. 

(4)  Verification  and  validation.  Verification  and 
validation  is  the  process  of  convincing  the 
developer  and  the  user  that  the  program  meets 
the  spec i ficaticns .  Two  complementary  techni¬ 
ques  are  used  to  verify  and  validate  the 
correctness  of  a  program:  (a)  proving,  and 

(b)  testing.  Program  proving  involves  con¬ 
structing  a  finite  sequence  of  logical  state¬ 
ments  ending  in  the  statement  to  be  proved 
(usually  the  output  specification).  Program 
testing,  on  the  other  hand,  involves  the  sym¬ 
bolic  or  physical  execution  of  a  set  of  test 
cases  with  the  intent  of  exposing  embedded 
faults,  if  any,  in  the  program.  Because  of 
the  complexity  of  current  software  applications, 
software  verification  is  the  most  tiring,  expen¬ 
sive  and  unpredictable  phase  of  software  develop- 
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ment.  Roughly,  40%-50%  of  total  development 
effort  is  spent  on  software  verification  and 
validation. 


(5)  Operation  and  maintenance.  Operation  and  main¬ 
tenance  refer  to  the  actual  usage  of  the  soft¬ 
ware  and  all  activities  pertaining  to  correc¬ 
tion  of  errors  during  operation,  upgrading  to 
maintain  compatibility  with  the  changing  environ¬ 
ment,  and  the  introduction  of  minor  or  major 
improvements  in  the  software. 

To  minimize  and  possibly  eliminate  the  problem  of  low 
quality  software,  software  engineers  emulated  the  statis¬ 
tical  standards  of  traditional  engineering  quality  control. 

A  number  of  software  metrics  and  models  have  been  proposed 
to  provide  numerical  measures  of  software  quality.  Some 
metrics  assign  numerical  weights  to  the  number  of  operands/ 
operators,  number  of  branches,  module  sizes,  number  of  GO  TO 
statements,  etc.  and  use  these  as  measures  of  software 
quality.  Sometimes,  a  mathematical  formula  is  used  to 
weigh  these  measures  and  obtain  a  single  measure  of  quality. 

Some  approaches  in  this  area  involve  correlating  or  regressing 
software  metrics  to  quality  measures  such  as  the  number  of 
observed  errors.  For  example,  studies  have  been  undertaken 
to  analyze  correlations  between  the  number  of  errors  de¬ 
tected,  size  of  software,  number  of  operators/operands,  etc. 

[BAS83 ] .  | 
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A  criticism  of  this  static  approach  is  the  fact  that 
the  acceptability  and  quality  of  the  implemented  software 


design  cannot  be  solely  determined  by  the  number  of 
operators/operands,  number  of  unconditional  branches, 
module  size  or  the  like.  Software  quality  assessment 
must  be  more  than  a  bean- counting  function.  Assessment 
of  software  quality  should,  theoretically,  take  into  ac¬ 
count  all  the  processes  that  go  into  the  design,  programming, 
verification  and  validation  of  the  software. 

Software  errors  and  their  impact  on  quality* 

The  imperfectness  of  existing  design,  coding,  veri¬ 
fication  and  validation  techniques,  as  well  as  imperfectness 
of  humans,  causes  omission  and  commission  of  errors  in  the 
software.  It  is  the  nature  and  number  of  these  errors  that 
effect  the  quality  of  the  produced  software. 

A  software  error  is  any  discrepancy  between  what  the 
software  can  do  versus  what  the  user  or  the  computing  en¬ 
vironment  (i.e.,  physical  machine,  operating  system, 
compilers,  etc.)  wants  it  to  do.  It  can  be  as  trivial  as 
a  syntax  or  semantic  error  or  as  complex  as  a  run-time, 
specification,  or  performance  error.  Run-time  errors, 
occurring  during  actual  program  execution,  may  be  in  the 
form  of  domain,  computational  (logic)  ,  or  non-termination 

* - 

The  terms  error  and  fault  are  sometimes  used  interchange¬ 
ably  in  this  report.  Definitions  of  selected  terms  in 
software  engineering  are  given  in  Appendix  f • 


errors.  Specification  errors,  occurring  as  a  result  of 
discrepancies  between  user  requirements  and  the  statement 
of  specifications,  can  be  due  to  incomplete,  inconsistent, 
or  ambiguous  specifications.  Performance  errors  exist 
whenever  a  discrepancy  exists  between  the  actual  performance 
(efficiency)  of  the  program  and  its  desired  or  specified 
performance.  (For  a  classification  and  description  of 
software  errors,  see  Appendix  A.) 

Software  errors  detected  late  in  the  software  develop¬ 
ment  process  are  much  more  expensive  to  eliminate  than  errors 
discovered  early  in  the  development  process.  It  is,  there¬ 
fore,  desirable  that  software  errors  which  directly  affect 
the  quality  of  software,  should  be  prevented  and  exposed  as 
early  in  the  software  development  process  as  possible. 
Basically,  there  are  two  complimentary  approaches  for 
achieving  this  objective: 

(1)  Reduce  or  prevent  the  number  of  committed  soft¬ 
ware  errors  through  better  requirement  analysis, 
specification  and  design  techniques,  and  better 
programming/ implement at ion  disciplines . 

(2)  Increase  error  exposure  through  improved  veri¬ 
fication  and  validation  techniques.  Verifica¬ 
tion  and  validation  (e.g.,  testing)  should  be 
distributed  throughout  the  development  process 
to  assure  early  exposure  of  errors.  Monitoring 
of  software  quality  should  be  done  at  each  phase 
of  software  development. 
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It  is  well  recognized  that  the  quality  of  software 
can  be  built  into  it  during  development  by  using  effective 
software  design  methods  for  coding  disciplines.  A  good 
design  almost  always  results  in  good  implementation,  but 
a  poor  design  almost  always  does  not.  A  number  of  design 
philosophies  are  becoming  popular  in  practice:  (1)  func¬ 
tional  decomposition  design,  e.g.,  functional  decomposi¬ 
tion  method,  softech  design  method,  top-down  design  method 
[BER81,  PET77,  GRI78];  (2)  data  flow  design,  e.g.  Constantine’s 
structured  design  method,  Myers'  composite  design  method 
[BER81,  MYE75].  and  (3}  data  structure  design,  e.g. 

Warnier/Orr  Logical  Construction  of  Programs  method, 

Jackson  design  method  (BEP.81,  ORR78,  JAC76J. 

Structured  programming,  or  programming  using  re¬ 
stricted  constructs  such  as  IF  THEN  (ELSE),  DO  WHILE, 

REPEAT  UNTIL,  etc.  and  avoidance  of  00  TO  statements,  is 
becoming  popular  as  an  effective  coding  discipline. 

Maximum  and  early  exposure  of  errors  during  the  soft¬ 
ware  development  process  requires  using  reliable  proving 
and  testing  strategies  at  the  right  phase  of  the  develop¬ 
ment  cycle.  However,  one  should  be  aware  that  the  program 
proving  and  program  testing  remain  imperfect  tools  for 
verifying  program  correctness.  Neither  one  of  them  can, 
in  practice,  guarantee  program  correctness.  Proving  and 
testing  should  not  be  viewed  as  competing  tools;  they  are, 
in  fact,  complementary  methods  for  decreasing  the  likeli¬ 
hood  of  program  failure  (G0077). 
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2 . 2  Software  Testing 

Software  testing  is  largely  a  heuristic  process.  There 
have  been  attempts  to  formulate  a  theoretical  foundation  of 
software  testing  (see  [G0077]).  The  basic  problem  of 
testing  is  to  find  a  test  selection  rule  that  constitutes 
a  reliable  test.  A  reliable  test  is  a  test  which  is  suffi¬ 
cient  for  verifying  the  program’s  correctness.  To  make  the 
testing  effort  practical  --  requiring  less  effort  and  cost 
to  use  one  would,  naturally,  pick  only  a  small  subset  of 
the  input  domain  that  hopefully  would  reveal  all  the  errors 
in  a  program.  Howden  [HOW76]  argued  that  this  is  impossible 
since  one  can  always  create  a  program  which  can  defeat  the 
test.  He  contented  himself  with  finding  a  testing  strategy 
that  is  reliable  for  a  subclass  of  programs.  It  is  well 
known  that  testing  can  only  reveal  the  presence  of  errors, 
never  their  absence. 

Deterministic  testing  techniques  can  be  further  grouped 
into  the  structure  dependent  and  the  structure  independent 
techniques.  Structure  dependent  or  white  box  testing  views 
a  program  as  a  directed  flow  graph  in  which  a  node  represents 
a  set  of  statements  and  an  edge  is  the  control  flow  of  the 
program.  Test  cases  are  generated  based  on  the  program's 
flow  graph.  Structure  independent  testing  or  black  box 
testing  generates  test  cases  based  on  the  specifications 
(functions)  of  the  program.  Testing  the  specified  func¬ 
tions  (.functional  testing)  of  a  program  can  be  a  partially 
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random  process  if  test  cases  are  generated  randomly  for 
each  function  in  the  program.  It  is  not  a  purely  random 
process  since  each  function  has  to  be  tested.  On  the  other 
hand,  if  the  test  cases  for  each  function  are  generated 
based  on  the  program  structure,  the  testing  process  becomes 
deterministic . 

A  connection  exists  between  structure  independent  and 
structure  dependent  testing.  The  program  paths  from  the 
program  flow  graph  are  really  nothing  but  partitions  of 
the  input  domain.  All  paths  leading  to  an  output  or  combi¬ 
nation  of  outputs  are  nothing  but  equivalent  partitions 
which  will  cause  the  execution  of  the  path. 

Popular  examples  for  structure  dependent  testing  are 
path  testing,  symbolic  testing,  domain  testing,  and  muta¬ 
tion  testing.  For  structure  independent  testing,  we  have 
equivalence  partitioning,  boundary  value  testing,  and 
cause-effect  graphing.  Path  testing  requires  that  all 
edge-edge  (path)  transitions  in  the  program  flow  granh 
be  executed.  A  less  stringent  test  criterion  is  branch 
testing  which  only  requires  coverage  of  all  the  edges 
(branches)  of  the  graph.  A  further  less  stringent  testing 
criterion  is  statement  testing  which  merely  requires  co¬ 
verage  of  all  the  n'des  of  the  program  graph. 

Symbolic  execution  utilizes  symbolic  input  to  come 
up  with  outputs  which  are  symbolic  expressions  of  the  in¬ 
puts.  Domain  testing  is  currently  limited  to  linear  pro¬ 
grams  because  of  the  difficulty,  in  general,  in  deriving 


test  cases  from  the  path  predicates  of  the  program.  The 
path  predicate  of  a  path  is  the  condition  that  a  set  of 
input  data  has  to  satisfy  in  order  for  a  path  to  be  tra¬ 
versed  at  run-time.  Mutation  testing  makes  a  series  of 
minor  changes  to  the  program,  creating  a  set  of  programs 
known  as  mutations.  Test  data  is  generated  to  cause 
every  inequivalent  mutation  to  give  incorrect  results  on 
some  input.  Equivalence  partitioning  partitions  the 
specified  input  domain  into  a  finite  number  of  equival¬ 
ence  classes.  Test  cases  are  then  derived  for  each  class 
pretending  that  the  test  case  is  representative  of  all 
members  of  that  class.  Boundary- value  testing  is  similar 
to  equivalence  partitioning  but  requires  that  one  or  more 
elements  from  an  equivalence  class  be  selected  such  that 
each  edge  of  the  equivalence  class  is  subjected  to  a  test. 
Cause-effect  graphing  uses  a  cause-effect  graph  to  generate 
test  cases.  It  determines  combinations  of  input  conditions 
which  map  to  a  specific  output  condition. 

A  practical  approach  for  verifying  the  correctness 
of  real  wcrld  software  would  be  a  combination  of  testing 
and  proving  coupled  with  the  aid  of  software  tools  such 
as:  debugging  packages,  program  instrumentation,  software/ 
hardware  monitors,  simulators,  compilers,  link  editors, 
static  and  dynamic  analyzers,  regression  test  systems, 
test  case  generators.  Currently,  this  is  the  only  viable 
approach  for  maximizing  the  exposure  of  embedded  errors 


in  software.  An  effective  software  verification  plan 
should  cover  the  whole  development  process  so  that  errors 
are  exposed  and  corrected  as  early  as  possible. 
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2.3  Software  Reliabilit 


A  quantifiable  measure  of  quality  that  has  become 
popular  in  software  engineering  practice  is  software  relia¬ 
bility.  It  was  necessitated  by  the  inability  of  existing 
software  verification  and  validation  techniques  to  guarantee 
correct  software. 

There  are  a  number  of  views  as  to  what  software 
reliability  is  and  how  it  should  be  quantified.  Some 
people  believe  that  this  measure  should  be  binary  in 
nature  so  that  an  incorrect  program  would  have  zero  relia¬ 
bility  while  a  perfect  program  would  have  a  reliability 
one.  This  view  parallels  that  of  program  proving  whereby 
the  program  is  either  correct  or  incorrect.  Others,  how¬ 
ever,  feel  that  software  reliability  should  be  defined  as 
the  relative  frequency  (or  percentage)  of  the  times  that 
the  program  works  as  intended  by  the  user.  This  view  is 
similar  to  that  taken  in  testing  where  a  percentage  of 
the  successful  cases  is  used  as  a  measure  of  program 
quality . 

According  to  the  latter  viewpoint,  software  relia¬ 
bility  is  a  probabilistic  measure  and  is  defined  as  the 
probability  that  a  software  error  which  causes  discrepan¬ 
cies  from  specified  requirements  in  a  specified  environment 
does  not  lead  to  a  failure  during  a  specified  exposure 
period.  Note  that  the  probabilistic  nature  of  this  measure 


is  due  to  the  uncertainty  in  the  usage  of  the  various  soft¬ 
ware  functions.  Such  discrepancies  are  also  known  as  soft¬ 
ware  faults.  The  specified  requirements  refer  to  the 
functional  requirements  desired  by  the  user.  Specified 
environment  means  that  the  software  need  be  correct  only 
for  its  specified  inputs  and  specified  computing  environ¬ 
ment.  The  specified  exposure  period  may  mean:  (1)  a  single 
run  or  a  number  of  runs;  or  (2)  time  unit  expressed  as  CPU 
time  units  or  calendar  time.  In  simple  terms,  if  a  user 
executes  a  software  product  several  times  (according  to 
the  distribution  of  his  needs)  and  95%  of  the  time  the 
software  provided  him  with  acceptable  (correct)  results, 
then  the  software  is  said  to  be  95%  reliable. 

A  more  precise  definition  of  software  reliability 
which  captures  the  points  mentioned  above  follows  [MOR83] 
Let  F.  be  a  class  of  faults,  defined  arbitrarily,  and  T  be 
a  measure  of  relevant  time,  the  units  of  which  are  dictated 
by  the  application  at  hand.  Then  the  reliability  of  the 
software  package  with  respect  to  the  class  of  faults  E  and 
with  respect  to  the  metric  T,  is  the  probability  that  no 
error  of  the  class  occurs  during  the  execution  of  the  pro¬ 
gram  for  a  prespecified  period  of  relevant  time. 

Assessment  of  software  reliability  can  be  a  non¬ 
trivial  process.  The  reasons  are  as  follows: 

(1)  Most  software  is  large  and  complex.  Embedded 

faults  may  not  be  easily  detectable  by  existing 
verification  and  validation  techniques. 
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(2)  Users  are  not  always  100  percent  certain  about 
their  requirements.  User  input  and  functional 
distributions  are  not  easily  predictable. 

(3)  Resources  (time  and  money)  allocated  for  soft¬ 
ware  development  are  always  limited;  hence,  the 
developer  may  not  have  enough  time  and  money  to 
test  for  all  possible  user  inputs. 

Granting  that  software  reliability  can  be  measured 
despite  the  above  obstacles,  a  logical  question  is  what 
purpose  does  it  serve.  Software  reliability  is  a  useful 
measure  in  planning  and  controlling  resources  (time  and 
money)  during  the  software  development  process  so  that  high 
quality  software  can  be  developed.  It  is  also  a  useful 
measure  for  giving  the  user  confidence  about  software  per¬ 
formance.  Planning  and  controlling  testing  resources  via 
the  software  reliability  measure  can  be  done  by  balancing 
the  additional  cost  of  testing  and  the  corresponding  im¬ 
provement  in  software  reliability.  As  more  and  more  errors 
are  exposed  by  the  testing  and  verification  process,  the 
additional  cost  of  exposing  the  remaining  errors  generally 
rises  very  quickly.  Thus,  there  is  a  point  beyond  which 
continuation  of  testing  to  further  improve  the  reliability 
of  software  can  be  justified  only  if  such  improvement  is 
cost  effective.  An  objective  measure  like  software  relia¬ 
bility  can  be  used  to  study  such  a  trade-off. 

The  current  approaches  for  measuring  software  relia¬ 
bility,  basically,  parallel  those  of  hardware  reliability 


assessment.  However,  appropriate  modifications  have  been 
made  before  extending  the  hardware  theory  to  software  to 
account  for  the  inherent  differences  between  software  and 
hardware.  Hardware  exhibits  mixtures  of  decreasing  and 
increasing  failure  rates.  The  decreasing  failure  rate  is 
due  to  the  fact  that,  as  time  on  the  hardware  system  accumu¬ 
lates,  failures  (most  probably  from  design  errors)  are  en¬ 
countered  and  the  errors  fixed.  The  increasing  failure 
rate  is  due  primarily  to  hardware  component  wearout  or 
aging.  There  is  no  such  thing  as  wearout  in  software. 

It  is  true  that  software  may  become  obsolete  because  of 
changes  in  the  user  and  computing  environment,  but  once 
we  modify  the  software  to  reflect  these  changes,  we  no 
longer  talk  of  the  same  software  but  of  an  enhanced  or 
modified  version.  Like  hardware,  software  exhibits  a 
decreasing  failure  rate  (improvement  in  quality)  as  the 
usage  time  on  the  system  accumulates  and  errors  (due  to 
design  and  coding)  are  fixed.  Thus,  a  hardware-based 
approach  to  software  reliability  assessment  can  be  used 
only  in  appropriate  environments. 

It  should  be  noted  that  an  assessed  value  of  the 
software  reliability  measure  is  always  relative  to  a  given 
user  environment.  Two  users  exercising  two  different  sets 
of  paths  in  the  same  software  may  have  different  values  of 
the  reliability  of  software. 

A  number  of  analytical  approaches  have  been  developed 
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to  address  the  problem  of  software  reliability  assessment. 
These  approaches  are  based  mainly  on  the  failure  history 
of  the  software.  They  may  be  divided  into  time-dependent 
and  time-independent  approaches.  The  time  dependent  ap¬ 
proach  is  based  on  either  times  between  software  failures 
or  on  failure  counts  in  specified  time  intervals.  The  time 
independent  approach  uses  either  fault  seeding  methods  or 
input  domain  analysis. 

In  the  time  dependent  approach,  the  times  between 
failures  or  the  number  of  failures  observed  in  a  sequence 
of  test  time  intervals  are  used  to  estimate  the  shape  of 
the  hypothesized  failure  (hazard)  rate  function.  From  the 
estimated  failure  rate  function,  one  can  estimate  the 
number  of  faults  remaining  in  the  software,  mean-time-to- 
failure  (MTTF) ,  and  software  reliability. 

In  the  fault- seeding  approach,  a  known  number  of 
faults  is  seeded  (planted)  in  the  program.  After  testing, 
the  numbers  of  exposed  seeded  and  indigenous  fat’ts  are 
counted.  Using  combinatorics  and  maximum  likelihood  esti¬ 
mation,  one  can  then  estimate  the  number  of  indigenous 
faults  in  the  program  and  also  the  reliability  of  the  pro¬ 
gram  . 

In  the  input  domain  based  models ,  the  procedure  is  to 
generate  a  set  of  test  cases  from  an  input  (operational) 
distribution.  The  difficulty  of  estimating  the  input  dis¬ 
tribution  is  overcome  by  partitioning  the  input  domain  into 


2-15 


a  set  of  equivalence  classes.  An  equivalence  class  is 
usually  associated  with  a  program  or  logic  path.  The  relia¬ 
bility  measure  is  then  calculated  from  the  observed  failures 
after  symbolically  or  physically  executing  the  generated 
test  cases. 

The  reliability  of  software  grows  as  it  evolves  in 
its  life  cycle.  Ver if ication  ‘ and  testing  should  be  per¬ 
formed  as  early  as  the  design  stage  to  expose  design  errors. 
If  possible,  the  reliability  of  the  design  should  also  be 
assessed.  Currently,  no  tool  or  model  is  available  to  pre¬ 
dict  the  reliability  of  the  software  as  early  as  the  design 
stage.  Testing,  to  expose  errors  after  the  design  phase, 
is  usually  done  in  stages.  The  first  stage  of  testing  is 
done  at  the  module  level  by  the  implementing  programmer. 
Modules  are  then  integrated  to  form  partial  or  the  whole 
system.  The  system  is  then  subjected  to  integration  testing 
(also  known  as  alpha  testing) .  Software  is  then  given  to 
several  "friendly  users"  who  are  willing  to  use  the  software 
in  an  operational  environment.  The  problems  encountered 
with  the  software  are  reported.  This  is  known  as  beta 
testing.  Finally,  software  is  released  to  users  and  cor¬ 
rections  are  issued  against  it  as  problems  are  reported 
by  users. 

Essentially,  this  overall  testing  process  makes  soft¬ 
ware  reliability  a  growth  process.  However,  the  reliability 
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of  software  can  decrease  as  a  result  of  the  software  correc¬ 
tion  (debugging)  process.  This  happens  when  additional 
errors  are  accidentally  injected  into  the  system  while 
removing  some  other  errors.  Switching  from  module  testing 
to  integration  testing  to  beta  testing  may  also  disturb  the 
perceived  software  reliability  growth  process.  A  temporary 
surge  of  exposed  errors  may  be  observed  when  we  switch  to 
different  test  strategies  during  the  software  development 
process.  The  use  of  better  design,  coding  and  verifica¬ 
tion  techniques,  coupled  with  effective  software  manage¬ 
ment  techniques,  would  reduce  the  likelihood  of  software 
reliability  deteriorating  over  its  life  cycle. 
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2 . 4  Models  for  Software  Reliability  Assessment 

A  number  of  models  have  been  proposed  during  the  last 
ten  years  for  assessing  software  reliability.  Most  of  them 
are  based  on  the  failure  history  of  software.  They  can  be 
classified  according  to  the  nature  of  the  failure  process 
studied  as  indicated  below: 

1)  Times  Between  Failures  (TBF)  Models 

In  this  class  of  models  the  process  under  study 
is  the  time  between  failures.  The  general  ap¬ 
proach  is  to  assume  that  the  time  between,  say, 
the  (i-l)st  and  the  ith  failures  follows  a  dis¬ 
tribution  whose  parameters  depend  on  the  number 
of  faults  remaining  in  the  program  during  this 
interval.  Estimates  of  the  parameters  are  ob¬ 
tained  from  the  observed  values  of  times  between 
failures.  Estimates  of  software  reliability 
mean  time  to  next  failure,  etc.  are  then  ob¬ 
tained  from  the  appropriate  e.uations. 

2)  Failure  Count  (FC)  Models 

The  interest  in  this  class  of  models  is  in  the 
number  of  failures  in  specified  time  intervals 
rather  than  in  the  times  between  failures.  The 
failure  counts  are  assumed  to  follow  a  known 
stochastic  process  with  a  time  dependent  dis¬ 
crete  or  continuous  failure  rate.  Parameters 
of  the  failure  rate  can  be  estimated  from  the 
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observed  values  of  failure  counts  or  from  failure 


times.  Estimates  of  software  reliability,  mean 
time  to  next  failure,  etc.  can  be  obtained  from 
the  relevant  equations. 

3)  Fault  Seeding  ( FS )  Models 

The  basic  approach  in  this  class  of  models  is  to 
"seed"  a  known  number  of  faults  in  a  program  which 
is  assumed  to  have  an  unknown  number  of  indigenous 
faults.  The  program  is  tested  and  the  observed 
numbers  of  exposed  seeded  and  indigenous  faults 
are  counted.  From  these,  an  estimate  of  the  fault 
content  of  the  program  prior  to  seeding  is  obtained 
and  used  to  assess  software  reliability,  etc. 

4)  Input  Domain  Based  (IDB)  Models 

The  basic  approach  taken  here  is  to  generate  a 
set  of  test  cases  from  an  input  distribution  which 
is  assumed  to  be  representative  of  the  operational 
usage  of  the  program.  Because  of  the  difficulty 
in  obtaining  this  distribution,  the  input  domain 
is  partitioned  into  a  set  of  equivalence  classes, 
each  of  which  is  usually  associated  with  a  program 
path.  An  estimate  of  program  reliability  is  ob¬ 
tained  from  the  failures  observed  during  physical 
or  symbolic  execution  of  the  test  cases  sampled 
from  the  input  domain. 

i  i 


Another  classification  of  the  models  in  (1)  and  (2) 
above  can  be  based  on  the  inference  viewpoint,  classical 
or  Bayesian.  However,  most  of  the  work  has  been  along 
classical  lines. 
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3.  TIMES  BETWEEN  FAILURES  (TBF)  MODELS 

This  is  one  of  the  earliest  classes  of  models  pro¬ 
posed  for  software  reliability.  When  interest  is  in 
modeling  times  between  failures,  it  is  expected  that  the 
successive  failure  times  will  get  longer  as  faults  are 
removed  from  the  software  system.  For  a  given  set  of  ob¬ 
served  values  this  may  not  be  exactly  so  due  to  the  fact 
that  failure  times  are  random  variables  and  observed  values 
are  subject  to  statistical  fluctuations. 

A  number  of  models  have  been  proposed  to  describe 
such  failures  in  a  software  system.  Let  a  random  variable 
Ta  denote  the  time  between  the  (i-l)st  and  the  ith  failures. 
Basically,  the  models  assume  that  T^  follows  a  known  distri¬ 
bution  whose  parameters  depend  on  the  number  of  faults  re¬ 
maining  in  the  system  after  the  (i-l)st  failure.  The 
assumed  distribution  is  supposed  to  reflect  the  improvement 
in  software  quality  as  faults  are  detected  and  removed  from 
the  system. 

Various  models  for  the  times  between  failures  phenomenon 
are  described  in  the  following  subsections.  A  detailed  de¬ 
scription  of  selected  models  is  given  in  Appendix  C. 

Note  that  the  models  described  below  differ  primarily 
in  their  treatment  of  the  nature  of  the  hazard  function 
associated  with  the  successive  software  failures. 
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as  hazard  rate  or 


The  hazard  function  (also  known 
failure  rate)  z(t)  is  defined  as  the  conditional  proba¬ 
bility  that  a  fault  is  exposed  in  the  interval  t  to  t  +  At , 
given  that  the  fault  did  not  occur  prior  to  time  t.  The 
reliability  function  R(t)  is  the  probability  that  no 
faults  will  occur  from  time  zero  to  time  t.  Also,  the 
functions  z(t)  and  R(t)  are  related  in  the  following 
form: 

z(t)  =  t-dR(t)/dt]/R(t) 
or  t 

P.(t)  =  exp (-|  z(x)dx) 

0 

Also,  mean- time-to-fai lure  (MTTF)  =  l/z(t). 

Estimation  of  reliability,  once  the  hazard  function 
z(t)  is  known  is  thus  straightforward. 

For  details  of  hazard  function  and  other  relevant 
reliability  concepts ,  see  Appendix  B. 


3 . 1  Jelinski  and  Moranda  De-eutrophication  Model  (Model  TBF1) 

This  is  one  of  the  earliest  and  probably  the  most  com¬ 
monly  used  models  for  assessing  software  reliability  [JEL72], 
It  assumes  that  there  are  N  software  faults  at  the  start  of 
testing,  each  is  independent  of  others  and  is  equally  likely 
to  cause  a  failure  during  testing.  A  detected  fault  is 
removed  with  certainty  in  negligible  time  and  no  new  faults 
are  introduced  during  the  debugging  process.  The  software 
failure  rate  or  the  hazard  function  at  any  time  is  assumed 
to  be  proportional  to  the  current  fault  content  of  the 
tested  program.  In  other  words,  the  hazard  function  during 
t^,  the  time  between  the  (i-l)st  and  ith  failures  is  given 
by 

Z  Ct±)  *  4>[N  -  (i-1)  ]  , 

where  $  is  a  proportionality  constant. 

Note  that  this  hazard  function  is  constant  between 
failures  but  decreases  in  steps  of  size  <f>  following  the 
removal  of  each  defect. 

A  typical  plot  of  the  hazard  function  for  N  =  100, 

<t>  -  .01  is  shown  in  Figure  3.1.  Details  of  this  model 
are  given  in  Appendix  C-l. 
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- ►CUMULATIVE  TIME 

Figure  3.1  A  Typical  Plot  of  Z(t.)  for  Model  TBF1 
(N  -  100,  4  -  .01) 


3 . 2  Schick  and  Wolverton  Linear  Model  (Model  TBF2) 

This  model  is  based  on  the  same  assumptions  as  the 
TBF1  model  except  that  the  hazard  function  at  any  time  is 
assumed  to  be  proportional  to  the  current  fault  content  of 
the  program  as  well  as  to  the  time  elapsed  since  the  last 
failure  [SCH73],  Under  these  assumptions  the  hazard  function 
z(O)  between  the  (i-l)st  and  the  ith  failures  is  given  by 

z(ti)  =  <HN  -  (i-DKj 

where  0  is  a  proportionality  constant  and  the  other  quantities 
are  as  defined  earlier. 

Note  that  in  some  papers  O  has  been  taken  to  be  the 
cumulative  time  from  the  beginning  of  testing.  That  interpre¬ 
tation  of  t^  seems  to  be  inconsistent  with  the  interpretation 
in  the  original  paper  [SCH73], 

We  note  that  the  above  hazard  rate  is  linear  with  time 
within  each  failure  interval,  returns  to  zero  at  the  occurrence 
of  a  failure  and  increases  linearly  again  but  at  a  reduced 
slope,  the  decrease  in  slope  being  proportional  to  0.  This 
behavior  for  N  =  150,  <p  =  .02  is  shown  in  Fig.  3.2,  Details 
of  this  model  are  given  in  Appendix  02. 

5.2.1  Schick  md  Wolverton  Parabolic  Model  (Model  TBF3) 

This  is  a  modification  of  Model  TBF2  [SCH78]  whereby  the 
hazard  function  is  assumed  to  be  parabolic  in  test  time  and 
is  given  by 


3-5 


z(ti)  =  <J>[N  -  (i-l)](-at?  +  bti  +  c) 
or 

z(t.)  =  $lc[N-  (i-1)  3  +  <j>  [N  -  (i-l)](-at?  +  bt.) 

where  a,b,c  are  constants  and  the  other  quantities  are  as 
defined  above. 

This  function  consists  of  two  components.  The  first 
is  basically  the  hazard  function  of  model  TBF1.  The  super¬ 
imposition  of  the  second  term  indicates  that  the  likelihood 
of  a  failure  occurring  increases  rapidly  as  the  test  time 
accumulates  within  a  testing  interval.  At  failure  times 
(xi=0),  the  hazard  function  is  proportional  to  that  of 
model  TBF1, 
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3. 3  Geometric  De-eutrophication  Model  (Model  TBF3) 

This  is  a  variation  of  the  TBF1  model  (3.1)  and  was 
proposed  by  Moranda  [MOR7S ,M0R81]  to  describe  testing  situ¬ 
ations  where  faults  are  not  removed  until  the  occurrence  of 
a  fatal  one  at  which  time  the  accumulated  group  of  faults 
is  removed.  In  such  a  situation,  the  hazard  function  after 
a  restart  can  be  assumed  to  be  a  fraction  of  the  rate  which 
attained  when  the  system  crashed.  For  this  model,  the 
hazard  function  during  the  ith  testing  interval  is  given  by 

2(ti)  =  Dk1'1, 

where 

D  is  the  fault  detection  rate  during  the  first  interval, 

and 

k  is  a  constant,  0  <  k  <  1. 

A  typical  plot  of  z(t^)  for  D  =  0.5  and  k  =  0.95 
is  shown  in  Figure  3.3.  Details  of  the  model  are  given 
in  Appendix  C- 3. 

3.3.1  Hybrid  Geometric  Poisson  Model  (Model  TBF4) 

This  model  was  proposed  by  Moranda  [MOR76]  as  a  candi¬ 
date  for  depicting  the  initial  segment  of  hardware  system 
testing.  It  covers  the  burn  in  and  steady  state  interval. 

It  is  a  composite  of  the  geometric  process  and  a  pure  Poisson 
model.  The  hazard  function  for  this  model  is 

z ( t i )  =  Dk1"1  +  e 

where  0  is  the  parameter  of  the  Poisson  process,  and  D  and K 
are  as  defined  above. 
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Figure  3.3  A  Typical  Plot  of  the  Hazard  Function 
for  the  Model  TBF3  (D  •  0.5,  k  -  0.95) 
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3-  4  Goel  and  Okumoto  Imperfect  Debugging  Model  (Model  TBF5) 


All  of  the  models  discussed  so  far  assume  that  the 
faults  are  removed  with  certainty  when  detected.  However, 
in  practice  [see  MIY7S,THA76]  that  is  not  always  the  case. 

To  overcome  this  limitation,  Goel  and  Okumoto  [GOE78b ,GOE78d , 
GOE79bJ  oroposed  an  imperfect  debugging  model  IIDM]  which 
is  basically  an  extension  of  model  TBF1  [JEL72]. 

In  this  model,  the  number  of  faults  in  the  system  at 
time  t,  X(t),  is  treated  as  a  Markov  process  whose  transi¬ 
tion  probabilities  are  governed  by  the  probability  of  im¬ 
perfect  debugging.  Times  between  the  transitions  of  X(t) 
are  taken  to  be  exponentially  distributed  with  rates  de¬ 
pendent  on  the  current  fault  content  of  the  system.  Ex¬ 
pressions  are  derived  for  performance  measures  such  as  the 
distribution  of  time  to  a  completely  debugged  system, 
distribution. 'of  the  number  of  remaining  faults,  and  soft¬ 
ware  reliability. 

For  this  model,  the  hazard  function  during  the  interval 
between  the  (i-l)st  and  the  ith  failures  is  given  by 


z(ti)  =  [N  -  p(i-l) ] A. 


where 

N 

is 

the 

initial  fault  content  of 

the  system, 

p 

is 

the 

probability  of  imperfect 

debugging , 

and 

A 

i  s 

the 

failure  rate  per  fault. 

3.5  Littlewood-Verrall  Bayesian  Model  (Model  TBF6; 

Littlewood  and  Verrall  [LIT731  took  a  different  ap¬ 
proach  to  the  development  of  a  model  for  times  between 
failures.  They  argued  that  software  reliability  should 
not  be  specified  in  terms  of  the  number  of  errors  in  the 
program.  Also,  they  adopted  a  subjective  approach  to  the 
treatment  of  failures  and  formulated  a  Bayesian  model. 

Specifically,  the  times  between  failures  are  assumed 
to  follow  an  exponential  distribution  but  the  parameter  of 
this  distribution  is  treated  as  a  random  variable  with  a 
gamma  distribution.  By  taking  different  forms  for  one 
of  the  parameters  of  this  gamma  distribution,  it  is  claimed 
that  the  failure  phenomena  in  different  environments  can 
be  explained  by  this  model. 
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4.  FAILURE  COUNT  MODELS 

This  class  of  models  is  concerned  with  modelling  the 
number  of  failures  in  given  testing  intervals.  As  faults 
are  removed  from  the  system,  it  is  expected  that  the  number 
of  failures  observed  per  unit  time  (for  any  reasonable  form 
or  units  of  the  measure  time)  will  decrease.  If  this  is  so, 
then  the  cumulative  number  of  failures  versus  time  curve 
will  eventually  level  off.  In  this  setup,  the  time  intervals 
may  be  fixed  a  priori  and  the  number  of  failures  in  each  in¬ 
terval  is  a  random  variable. 

Several  models  have  been  suggested  to  describe  this 
failure  phenomenon.  The  basic  idea  behind  most  of  these 
models  is  that  of  a  Poisson  distribution  whose  parameter 
takes  different  forms  for  different  models.  It  should  be 
noted  that  Poisson  distribution  has  been  found  to  be  an  ex¬ 
cellent  model  in  many  fields  of  application  where  interest 
is  in  the  number  of  occurrences  of  some  quantity  of  interest. 

One  of  the  earliest  models  in  this  category  was  proposed 
by  Shooman  [SH072],  Taking  a  somewhat  similar  approach, 

Musa  (MIJS75]  later  proposed  another  failure  count  model  based 
on  execution  time.  Schneidewind  [SCH75]  took  a  different 
approach  and  studied  the  fault  counts  over  a  series  of  time 
intervals.  Coel  and  Okumoto  [GOE79a]  introduced  a  time  dependent 
failure  rate  of  the  underlying  Poisson  process  and  developed 

i 
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the  necessary  analytical  details  of  the  models.  Several 
other  models  have  also  been  proposed  in  this  class,  mostly 
as  extensions  of  the  corresponding  time  between  failure 
models . 

A  brief  description  of  those  models  in  this  category 
is  given  below.  Details  of  selected  models  are  presented 
in  Appendix  D. 
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4 . 1  Goel-Okumoto  Non-Homogeneous  Poisson 
Process  Model  (Model  FC1) 

In  this  model  Goel  and  Okumoto  [GOE79a]  basically  claimed 
that  a  software  system  is  subject  to  failures  at  random  times 
caused  by  faults  present  in  the  system.  Letting  N(t)  be  the 
cumulative  number  of  failures  observed  by  time  t,  they  pro¬ 
posed  that  N(t)  can  be  modelled  as  a  non-homogeneous  Poisson 
process  (NHPP) ,  i.e.  as  a  Poisson  process  with  a  time  dependent 
failure  rate.  Based  on  their  study  of  actual  failure  data  from 
many  systems,  they  proposed  the  following  form  of  the  model 

P(N(t)  =  y}  =  liUCpI  e‘m(t),  y  =  0,1,2,  ... 

where 

m(t)  -  a(l  -  e~bt) 

also  . 

X(t)  *  m'(t)  *  abe" 

In  the  above  m(t)  is  the  expected  number  of  failures  detected 
by  time  t  and  A(t)  is  the  failure  rate.  A  typical  plot  of  the 
X(t)  function  is  shown  in  Figure  4.1. 

In  this  model  a  is  the  expected  number  of  failures  to 
be  observed  eventually  and  b  is  the  fault  detection  rate  per 
fault.  It  should  be  noted  that  in  this  model  the  number  of 
faults  to  be  observed  is  treated  as  a  random  variable  whose 
observed  value  depends  on  the  test  environment.  This  is  a 
basic  departure  from  most  of  the  other  models  which  treat 
the  number  of  faults  to  be  a  fixed  unknown  constant. 

Details  of  this  model  are  given  in  Appendix  D-l. 


Figure  4.1  A  Typical  Plot  for  the  Failure  Rate  Function 
for  Model  FC1  (a  -  175,  b  -  0.05) 


4.1.1  Schneidewind  Model  (Model  FC2) 

Using  a  different  approach  than  described  above 
Schneidewind  [SCH75]  studied  the  number  of  faults  detected 
during  a  time  interval  and  failure  counts  over  a  series  of 
time  intervals.  He  assumed  that  the  failure  process  is  a 
non-homogeneous  Poisson  process  with  an  exponentially  de¬ 
caying  intensity  function 

d(i)  -  ae-3*,  a, 3  >  0,  i-1,2,  ... 


4-5 


4 . 2  Goel  Modified  Non-Homogeneous  Poisson 
Process  Model  (Model  Fc5J 

Most  of  the  times  between  failures  and  failure  count 
models  assume  that  a  software  system  exhibits  a  decreasing 
failure  rate  pattern  during  testing.  In  other  words,  they 
assume  that  software  continues  continues  to  be  fault  free 
as  testing  progresses. 

In  practice,  it  has  been  observed  that  in  many  testing 
situations,  the  failure  rate  (number  of  failures  per  unit 
time)  first  increases  and  then  decreases.  In  order  to  model 
this  increasing/decreasing  failure  rate  process,  Goel  [GOE82] 
proposed  the  following  modified  version  of  the  NHPP  model 
(Model  FC1) . 

p{N(t )  -y)  •  Wf??7 

where 

m(t)  *  a(l  -  e’bt  ) 

a  is  expected  number  of  faults  to  be  eventually 
detected 

b  and  c  are  constants  that  reflect  the  severity  of 
testing 

The  failure  rate  is  given  by 

A(t)  -  m'(t)  -  abc  e‘btCtc'1 

A  typical  plot  of  the  A(t)  function  is  shown  in  Figure 
4.2.  Details  of  the  model  are  given  in  Appendix  D-2. 
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Figure  4.2  A  Typical  Plot  of  the  Failure  Rate  Function 
for  Model  FC3  (a  ■  500 r  b  -  0.015,  c  -  1.5) 


4. 3  Musa  Execution  Time  Model  (Model  FC4) 

In  this  model  Musa  [MUS75]  makes  assumptions  that  are 
similar  to  those  of  the  model  TBF1  of  Jelinski  and  Moranda 
except  that  the  process  modelled  is  the  number  of  failures 
in  specified  execution  time  intervals.  The  hazard  function 
for  this  model  is  given  by 

z(x)  =  $f (N  -  nc) 

t  -  execution  time  utilized  in  executing  the  program 
up  to  the  present 

f  *  linear  execution  frequency  (average  instruction 
execution  rate  divided  by  the  number  of  instruc¬ 
tions  in  the  program) 

<t>  -  proportionality  constant,  which  is  a  fault  expo¬ 
sure  ratio  that  relates  fault  exposure  frequency 
to  the  linear  execution  frequency 
nc  -  number  of  faults  corrected  during  (0,r) 

One  of  the  main  features  of  this  model  is  that  it 
explicitly  emphasizes  the  dependence  of  the  hazard  function 
on  execution  time.  Musa  also  provides  a  syst  natic  approach 
to  converting  the  model  so  that  it  can  be  appli  ..ole  or 
calendar  time. 

Details  of  the  model  are  given  in  Appendix  D-3. 
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4 . 4  Shopman  Exponential  Model  (Model  FC5) 

This  model  is  essentially  similar  to  model  TBF1  of 
Jelinski  and  Moranda.  For  this  model  the  hazard  function 
[SH072 ]  is  of  the  following  form 

z(t)  -  k[y  -  nc(x)  ] 


where 


t  -  operating  time  of  the  system  measured  from 
its  initial  activation 

I  -  total  number  of  instructions  in  the  program 
t  -  debugging  time  since  the  start  of  system 
integration 

nc(T)  -  total  number  of  faults  corrected  during  t, 
normalized  with  respect  to  I,  and 
k  -  proportionality  constant 


Details  of  this  model  are  given  in  Appendix  D-4. 


4. 5  Geometric  Poisson  Model  (Model  FC6) 

Moranda  [MOR75a]  proposed  this  model  to  explain  the 
failure  phenomena  where  fault  occurrence  data  are  reported 
only  periodically  and  not  after  each  failure.  He  used  the 
Poisson  distribution  as  a  description  of  the  number  of 
faults  detected  in  a  fixed  period  of  time.  The  hazard 
function  during  the  ith  testing  interval  is  given  by 

z(t.)  =  AK1'1 

where 

A  -  average  number  of  faults  occuring  in  the 
first  interval 
K  -  a  constant,  0  <  K  <  1 

Note  that  this  is  a  discrete  version  of  the  NHPP  model 
of  Goel  and  Okumoto  (Model  FC3) .  Details  of  this  model  are 
described  in  Appendix  D-5. 
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4 . 6  Modified  Jelinski -Moranda  Model  (Model  FC7) 

As  the  name  implies,  this  model  is  a  modification  of 
model  TBF1  [see  SUK76],  It  was  proposed  to  explain  failur 
phenomena  where  more  than  one  fault  occurs  in  a  specified 
testing  interval.  The  hazard  function  for  the  ith  testing 
interval  is  given  by 

z(ti)  =  <t>[N  -  Mi.1] 

where 

N  -  total  number  of  faults  in  the  system 
-  total  number  of  faults  removed  up  to  the 
end  of  the  previous  testing  interval 
<j>  -  proportionality  constant 

Details  of  this  model  are  given  in  Appendix  D-6. 
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4. 7  Modified  Geometric  De-Eutrophication 
Model  (Model  FC81 

This  is  a  modification  [SUK76]  of  the  Geometric  De- 
Eutrophication  model  (Model  TBF5)  of  Moranda.  The  hazard 
function  during  a  testing  interval  is  a  constant  whose 
value  changes  at  the  beginning  of  the  next  testing  inter¬ 
val,  and  is  given  by 

M.  , 

z(ti)  -  Dk  1-1 

where 

D  =  fault  detection  rate  during  the  first 
testing  interval  ^ 
k  =  a  positive  constant  less  than  1 
Mi-1  *  cumulative  number  of  faults  detected  up 

to  the  end  of  the  (i-l)st  testing  interval. 

Details  of  this  model  are  given  in  Appendix  D-7. 


4 . 8  Modified  Schick  and  Wolverton  Model  (Model  FC9) 

In  this  model  the  faults  are  assumed  to  occur  inde¬ 
pendently  of  each  other.  The  fault  occurrence  rate  during  a 
testing  interval  is  proportional  to  the  number  of  faults 
remaining  in  the  system  at  the  beginning  of  this  interval 
and  to  the  total  time  previously  spent  in  testing  (including 
an  'averaged'  fault  search  time  during  the  current  time). 

The  expected  number  of  faults  occurring  during  the  ith 
interval  of  length  t^  is  then  given  by 

ElN.j  =  *[N-Mi>1l(T.,1  +  t., 

where 

^  is  the  total  number  of  faults  removed  up  to  the 
end  of  the  (i-l)st  interval, 
t^  is  the  ith  testing  interval 
T^j  is  the  cumulative  test  time  through  the 
(i-l)st  interval,  and 
<p  is  a  proportionality  constant 

A  detailed  description  of  this  model  is  given  in  Appendix 
D-8. 
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1 . 9  Generalized  Poisson  Model  (Model  FC  10) 

This  is  a  variation  of  the  NHPP  model  of  Goel  and 
Okumoto  (Model  FC1)  and  assumes  a  mean  value  function 

m(t.)  *  ♦(N-Mi_1)t® 

where 

-  total  number  of  faults  removed  up  to 
the  end  of  the  (i-l)st  debugging  interval 
<j>  -  constant  of  proportionality 
a  -  constant  used  to  rescale  time  t^ 

Details  of  this  model  are  given  in  Appendix  D-9. 
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4.10  IBM  Binomial  Model  (Model  FC111 

In  this  model  Brooks  and  Motley  [BR083]  consider  the 
fault  detection  process  in  software  testing  as  a  discrete 
process.  The  software  system  is  assumed  to  be  developed  and 
tested  incrementally.  This  means  that  some  modules  of  the 
system  may  be  available  for  testing,  or  in  and  out  of  test, 
while  others  are  not.  The  testing  phase  of  the  software 
system  is  assumed  to  consist  of  a  number  of  test  occa¬ 
sions  and  each  test  occasion  is  further  assumed  to  consist 
of  unit  intervals  of  testing.  Thus,  testing  effort  (i.e.  total 
number  of  unit  intervals)  in  each  test  occasion  could  be  dif¬ 
ferent.  This  model  also  assumes  that  faults  could  be  intro¬ 
duced  into  the  software  during  the  fault  removal  process. 

This  model  can  be  applied  at  the  module  level  or  at 
the  system  level.  If  the  model  is  applied  at  the  module  level, 
then  the  observed  process  is  the  fault  occurence  process  of 
each  test  occasion  of  the  module  under  test.  According  to 
this  model  the  number  of  faults  detected  during  the  ith  test 
occasion  in  module  j,  n.^,  follows  a  binomial  distribution  with 
parameters  N—  and  that  is 


Nij. 


P(nij  -xij>  - 


where,  N.j  *  expected  number  of  faults  remaining  in  module 
j  at  the  beginning  of  the  ith  test  occasion 
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n^j  *  number  of  faults  detected  in  module  j 
during  the  ith  test  occasion 
=  fault  detection  probability  for  the  ith  test 
occasion  from  module  j . 

If  the  model  is  applied  at  the  system  level,  then  the  ob¬ 
served  process  is  the  fault  occurrence  process  of  each  test 
occasion  of  the  software  system.  In  that  case,  the  number 
of  faults  detected  during  the  ith  test  occasion  of  the  soft¬ 
ware  system  follows  a  binomial  distribution  with  parameters 
fT  and  q^ ,  that  is 

Ni  x.  N-x. 

P<ni=xi>  =  Q  «i  • 

where,  *  expected  number  of  faults  remaining  at  the  be¬ 

ginning  of  the  ith  test  occasion. 
ni  *  number  of  faults  detected  during  the  ith  test  occa¬ 
sion  . 

“  fault  detection  probability  for  the  ith  test 
occasion  . 

Note  that  N.  ■  7  N.. 

1  jSJt  13 

where,  is  the  set  of  modules  tested  on  occasion  i. 

Details  of  this  model  are  given  in  Appendix  D-10. 
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4.11  IBM  Poisson  Model  (Model  FC12] 


The  assumptions  of  this  model  are  similar  to  those  of 
model  Fell  except  that  the  number  of  faults  found  during 
a  test  occasion  is  assumed  to  follow  a  Poisson  distribu¬ 
tion. 


This  model  also  can  be  applied  at  the  module  or 
the  system  level.  If  the  model  is  applied  at  the  module 
level,  then  the  observed  process  is  the  fault  occurrence  pro¬ 
cess  of  each  test  occasion  of  the  module  under  test.  According 
to  this  model  the  number  of  faults  detected  during  the  ith 
test  occasion  in  module  j  follows  a  Poisson  distribution  with 
parameter  ,  that  is. 


-N.  • 

«  1J  1J  (St1«i,) 
P{ntj  H — *- 


Xij 


where,  *  expected  number  of  remaining  faults  in  module  j 
at  the  beginning  of  test  occasion  i. 
n^j  ■  number  of  faults  detected  in  module  j  during 
test  occasion  i 

<j>„  ■  the  proportionality  factor  between  remaining 
faults  and  fault  detection  rate. 

^ii^ii  =  expected  number  of  faults  detected  in  module  j 
during  test  occasion  i. 


If  the  model  is  applied  at  the  system  level,  then  the  observed 
process  is  the  fault  occurrence  process  of  each  test  occasion 
of  the  software  system.  Then,  in  that  case,  the  number 
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of  faults  detected  during  the  ith  test  occasion  follows 
a  Poisson  distribution  with  parameter  Ni<j>i »  that  is 


’Mi 


P<ni-Xi> 


(N^) 


V 


where,  =  expected  number  of  remaining  faults  at  the 
beginning  of  the  ith  test  occasion, 
n^  a  number  of  faults  detected  during  the  ith  test 
occasion  . 

4k  *  the  proportionality  factor  between  remaining 
faults  and  fault  detection  rate. 

^i^i  *  expected  number  of  faults  detected  in  the  system 
during  test  occasion  i. 

Note  that  N .  *  .  Z  T  N.  , 
x 


where  is  the  set  of  modules  tested  on  occasion  i. 

Details  of  this  model  are  given  in  Appendix  D-ll. 
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5.  COMBINATORIAL  MODELS  (FS  AND  IDB  MODELS] 

In  this  section  we  give  a  brief  description  of  time- 
independent  models  that  have  been  proposed  for  assessing 
software  reliability.  As  mentioned  earlier,  the  two  ap¬ 
proaches  proposed  for  this  class  of  models  are  fault  seeding 
and  input  domain  analysis. 

In  fault  seeding  models  a  known  number  of  faults  is 
seeded  (planted)  in  the  program.  After  testing,  the  numbers 
of  exposed  seeded  and  indigenous  faults  are  counted.  Using 
combinatorics  and  maximum  likelihood  estimation,  one  can  then 
estimate  the  number  of  indigenous  faults  in  the  program  or 
the  reliability  of  the  software. 

The  basic  approach  in  the  input  domain  based  models  is 
to  generate  a  set  of  test  cases  from  an  input  (operational) 
distribution.  Because  of  the  difficulty  in  estimating  the 
input  distribution,  the  various  models  in  this  group  parti¬ 
tion  the  input  domain  into  a  set  of  equivalence  classes.  An 
equivalence  class  is  usually  associated  with  a  program  path. 
The  reliability  measure  is  calculated  from  the  observed 
failures  after  execution  (symbolic  or  physical)  of  the  sampled 
test  cases. 

The  fault  seeding  model  is  discussed  in  Section  5.1. 
Input  domain  based  models  are  described  in  Section  5.2.  More 
details  about  these  models  are  given  in  Appendix  E. 
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5.1  Mills  Seeding  Model  (FS1) 

A  number  of  models  have  been  proposed  but  the  most 
popular  (and  most  basic)  is  Mills'  Hypergeometric  model 
[MIL72].  This  model  requires  that  a  number  of  known  faults 
be  randomly  inserted  (seeded)  in  the  program  to  be  tested. 

The  program  is  then  tested  for  some  amount  of  time.  The 
number  of  original  indigenous  faults  can  be  estimated 
from  the  numbers  of  indigenous  and  seeded  faults  uncovered 
during  the  test  by  using  the  hypergeometric  distribution. 

The  procedure  adopted  in  this  model  is  similar  to 
the  one  used  for  estimating  population  of  fish  in  a  pond 
or  for  estimating  wildlife.  These  models  are  also  referred 
to  as  tagging  models  since  a  given  fault  is  tagged  as  seeded 
or  indigenous. 

The  relevant  formulae  are  given  in  Appendix  E.l. 

Lipow  [ LIP72]  modified  this  problem  by  taking  into  con¬ 
sideration  the  probability,  q,  of  finding  a  fault  (of  either 
kind)  in  any  test  of  the  software.  Then,  for  N  statistically 
independent  tests  the  probability  of  finding  x^  indigenous 

faults  and  x  seeded  faults  can  be  calculated  as  shown  in 
s 

Appendix  E- 1* 

Basin  { BAS 7  4]  suggested  a  two  stage  procedure  with  the 
use  of  two  programmers.  Estimates  of  the  number  of  indigenous 
faults  using  this  procedure  can  be  obtained  as  shown  in 
Appendix  E-l. 
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5 . 2  Input  Domain  Based  Models  (IDB  Models) 

Representative  models  in  this  class  are  those  proposed 
by  Nelson  [NEL78 ,BR075 ] ,  Ho  [H078]  and  Ramamoorthy  and 
Bastani  [RAM82 ] . 

5.2.1  Nelson  Model  (IDB1  Model) 

The  reliability  of  the  software  is  measured  by  exposing 
(running)  the  software  with  a  sample  of  n  inputs.  The  n 
inputs  are  randomly  chosen  from  the  input  domain  set  E  = 

(E^:  i  *  1,N)  where  each  E^  is  the  set  of  data  values  needed 
to  make  a  run.  The  random  sampling  of  n  inputs  is  done 
according  to  a  probability  distribution  P^;  the  set  (P^:  i  =  1,N) 
is  the  "operational  profile"  or  simply  user  input  distribu¬ 
tion.  If  ng  is  the  number  of  inputs  that  resulted  in  execu¬ 
tion  failures,  than  an  unbiased  estimate  of  software 
reliability  ft^is  1-  (ne/h).  However,  it  may  be  the  case  that 
the  test  set  used  during  the  verification  phase  may  not  be 
representative  of  the  expected  operational  usage.  Brown  and 

/V 

Lipow  [BR075]  suggested  an  alternative  formula  for  R  which  is 

^  N  f; 


*2  =  1  -  l  -1  P(E .  ) 

i-1  "j  J 


where 


n^  -  number  of  runs  sampled  from  input  subdomain 

f.  -  number  of  failures  observed  out  of  n.  runs. 

J  J 

The  main  difference  between  Nelson's  ft  and  Brown  and  Lipow's 
is  that  the  former  explicitly  incorporates  the  usage  dis¬ 
tribution  or  the  test  case  distribution  while  the  latter 
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implicitly  assumes  that  the  accomplished  testing  is 
representative  of  the  expected  usage  distribution.  Both 


i 

models  assume  prior  knowledge  of  the  operational  usage  I 

distribution. 


5.2.2  Ho  Model  (IDB2  Model) 


Reliability  estimation  in  this  model  proceeds  by  first 
generating  the  symbolic  execution  tree  of  the  program.  This 
tree  characterizes  all  the  execution  paths  and  their  associ¬ 
ated  outputs  in  the  program.  The  nodes  represent  statements 
while  the  edges  represent  the  state  vector  resulting  from 
symbolic  execution  along  the  path  from  the  root  statement 
to  the  current  statement.  A  procedure  for  generating  the 
symbolic  execution  tree  [H078]  is  given  below: 

I,  The  first  statement  is  the  root  of  the  tree. 

II.  If  a  leaf  is  not  a  STOP  or  RETURN  statement, 

symbolically  execute  the  statement  corresponding 
to  the  node.  If  the  current  statement  is  a  con¬ 
ditional  statement,  the  feasibility  of  the 
branches  is  examined,  New  nodes  are  created 
for  statements  which  are  successors  of  the 
current  statement.  Edges,  labelled  with  state 
vectors,  are  joined  between  the  current  node  and 
the  new  node (s ) . 

III.  Go  to  II. 

The  generated  execution  paths  from  the  symbolic  execu¬ 
tion  tree  are  proven  correct  or  are  sample  tested.  For  a 
given  path,  say  path  i,  if  it  is  proven  correct,  then  the 
path  reliability  R.  =  1 .  If  path  i  cannot  be  proven  correct, 
a  random  sample  of  N  test  cases  is  generated  that  will  exe¬ 
cute  path  i.  If  no  failures  result  from  the  execution  of  the 
N  test  cases,  then  R^  is  bounded  below  by  1  -  where 


is  the  confidence  interval  of  path  i.  The  length  of  is 

a  function  of  our  given  confidence  coefficient  a.  On  the 

other  hand,  if  n  failures  are  observed  and  the  errors  not 

N  -  n 

corrected,  then  is  bounded  below  by  — ^ —  -  C^.  ^  the 

observed  n  failures  are  corrected,  then  the  sample  testing 
is  repeated  for  path  i. 

A 

Finally,  the  software  reliability  estimate  R  is 


obtained  as 


ft  = 


m 

l 

i  =  1 


fiRi 


where : 

f^  -  weighting  factor  Qf  path  i  which  corresponds 
to  the  execution  frequency  of  path  i. 
m  -  total  number  of  execution  paths. 

One  difficulty  with  applying  this  approach  is  the 
large  number  of  paths  that  may  exist  in  any  large  size  software. 
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5.2.3  Ramamoorthy  and  Bastani  Model  (Model  IDB3) 

This  input  domain  based  model  estimates  the  relia- 

A 

bility  R  from  the  relation 

ft  =  1  -  V 

er 

where 

V  -  the  total  fault  size  remaining  in  the  program. 
er 

V  can  be  determined  by  testing  the  program  and  locating  and 
r 

estimating  the  size  of  faults  found.  A  fault  has  a  large 
size  if  it  is  easily  detected  (i.e.,  if  it  affects  many  in¬ 
put  elements).  A  fault  has  a  small  size  if  it  is  relatively 
difficult  to  detect.  The  size  of  a  fault  depends  on  the  way 
test  inputs  are  selected.  Good  test  case  selection  strate¬ 
gies  like  path  testing,  boundary  value  analysis,  magnify 
the  size  of  a  fault  since  they  exercise  error-prone  constructs. 
The  observed  fault  size  is  lower  if  random  testing  is  em¬ 
ployed.  Although  the  model  does  not  assume  random  testing 
(in  fact,  any  test  strategy  can  be  employed),  it  offers  no 
easy  or  systematic  way  to  estimate  ^  . 
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6.  ASSUMPTIONS ,  LIMITATIONS  AND 
APPLICABILITY  OF  MODELS 


In  this  section  we  discuss  the  appropriateness  or  other¬ 
wise  of  the  various  assumptions  underlying  the  models  of 
Sections  3  through  5  as  well  as  their  applicability  during 
software  development  phases.  Details  of  the  assumptions  have 
been  brought  out  in  Appendices  C,  D,  and  E  along  with  the 
details  of  the  models.  Not  all  the  assumptions  discussed 
here  are  relevant  to  any  given  model  but  as  a  totality,  they 
give  a  picture  of  the  kind  of  limitations  imposed  on  the 
use  of  software  reliability  models.  The  purpose  of  the 
following  discussion  is  to  focus  attention  on  the  framework 
within  which  the  existing  models  have  been  developed. 

It  should  be  pointed  out  that  the  arguments  presented 
here  regarding  the  assumptions  or  the  applicability  of  the 
models  may  not  be  universally  acceptable.  This  is  so  because 
the  software  development  process  is  very  environment  dependent. 
What  holds  true  in  one  environment  may  not  be  true  in  another. 
Because  of  this,  assumptions  that  are  reasonable,  e.g.  during 
the  testing  of  one  function  or  system  may  not  hold  true  in 
even  a  subsequent  test  phase  of  the  same  function  or  system. 

The  ultimate  decision  about  the  appropriateness  of  the  under¬ 
lying  assumptions  and  the  applicability  of  the  models  will 
have  to  be  made  by  the  user  of  a  model.  What  is  presented 
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here  should  be  helpful  in  determining  whether  the  assump¬ 
tions  underlying  a  given  model  are  representative  of  his 
testing  environment  and  in  deciding  which  model,  if  any, 
to  use. 

The  assumptions  are  discussed  one  at  a  time  in  Section 
6.1  along  with  a  list  of  the  models  which  use  the  assumption 
being  discussed.  Software  development  phases  and  the  appli¬ 
cability  of  models  in  each  phase  are  covered  in  Section  6.2. 
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6.1  Assumptions  and  limitations 


6.1.1  Independent  Times  Between  failures 

This  assumption  requires  that  the  times  between 
successive  failures  be  independent  of  each  other.  In 
general,  this  would  be  the  case  if  successive  test  cases 
were  independent,  i.e.,  were  chosen  randomly.  However, 
testing,  especially  functional  testing,  is  not  based  on 
independent  test  cases,  l.e.,  the  test  process  in  general 
is  not  a  random  process.  The  time  (or  the  additional 
number  of  test  cases)  to  the  next  failure  may  very  well 
depend  on  the  niture  or  time  c-f  the  previous  fault.  If 
a  critical  fault  is  uncovered,  the  tester  may  decide  to 
intensify  the  testing  process  and  look  for  more  potential 
critical  faults.  This  in  turn  may  mean  shorter  time  to 
the  next  failure. 

This  assumption  is  used  in  models  TBF1 ,  TBF2, 
TBF3 ,  TBF4 ,  TBF5  and  TBF6 . 
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6.1.2  A  Detected  Fault  Is  Immediately  Corrected 


The  models  that  require  this  assumption  assume 
that  the  software  system  goes  through  a  purification  process 
as  testing  uncovers  faults  which  are  immediately  removed. 

An  argument  can  be  made  that  this  assumption  is  implicitly 
satisfied  in  most  testing  environments.  When  a  fault  is 
detected,  the  testing  process  in  general,  but  not  always, 
can  proceed  without  removing  the  fault.  The  future  failure 
process  can  then  be  assumed  to  be  based  on  the  assumption 
that  the  fault  was  in  fact  removed  after  its  occurrence. 

If,  however,  the  fault  is  in  the  path  that  must  be 
tested  further,  this  assumption  would  be  satisfied  only 
if  the  fault  is  removed  prior  to  proceeding  with  the  re¬ 
mainder  of  the  test  bucket  or  the  test  cases  are  generated 
to  get  around  it. 

This  assumption  is  used  in  models  TBF1 ,  TBF2 ,  TBF3, 
TBF4,  TBF6 . 


6.1.3  No  New  Faults  are  Introduced  during 
the  Fault  Removal  Process 

The  purpose  of  this  assumption  is  to  ensure  that  the 
modelled  failure  process  does  have  a  monotonic  pattern. 

That  is,  the  subsequent  faults  are  exposed  from  a  system 
that  has  less  faults  than  before.  In  general,  this  may 
not  be  true  if  the  correction  process  follows  the  occurrence 
of  every  failure,  because  during  the  correction  process, 
other  paths  may  have  been  affected  leading  to  additional 
faults  in  the  system.  It  is  generally  considered  to  be  a 
restrictive  assumption  in  reliability  models.  The  only  way 
to  satisfy  this  is  to  ensure  that  the  correction  process 
does  not  introduce  new  faults. 

This  assumption  is  used  in  all  reliability  models. 


6.1.4  Failure  Rate  is  Proportional  to  the 
Number  of  Remaining  Faults 


This  assumption  is  the  same  as  saying  that  each 
remaining  fault  has  the  same  chance  of  being  detected  in 
a  given  testing  interval  between  failures.  This  assump¬ 
tion  is  a  reasonable  one  if  the  test  cases  are  chosen  to 
ensure  equal  probability  of  executing  all  portions  of  the 
code.  However,  if  one  portion  (or  a  set  of  paths)  is 
executed  more  thoroughly  than  another,  the  faults  in  the 
former  are  more  likely  to  be  detected  than  in  the  latter. 
Faults  residing  in  the  unreachable  (or  never  tested)  portion 
of  the  code  will  obviously  have  a  low,  or  zero,  probability 
of  being  detected. 

This  assumption  is  used  in  models  TBF1,  TBF2,  and  TBF4. 


6.1.5  Failure  Rate  Decreases  with  Test  Time 

This  assumption  implies  that  the  software  gets  better 
with  testing.  This  seems  to  be  a  reasonable  assumption  and 
can  be  justified  as  follows.  As  testing  proceeds,  faults 
are  detected.  They  are  either  removed  before  testing  con¬ 
tinues  or  they  are  not  removed  and  the  test  coverage  is 
narrowed  in  subsequent  tests.  In  the  former  case  the  sub¬ 
sequent  failure  rate  decreases  explicitly.  In  the  latter 
case,  the  failure  rate  (based  upon  the  entire  program)  de¬ 
creases  implicitly  as  smaller  and  smaller  portions  of  the 
code  are  subjected  to  testing. 

This  assumption  is  used  in  models  FC1,  FC3,  FC4,  and 


6.1.6  Increasing  Failure  Rate  Between  Failures 

This  assumption  implies  that  the  likelihood  of  finding 
a  fault  increases  as  the  testing  time  increases  within  a 
given  failure  interval.  This  would  be  a  justifiable  assump¬ 
tion  if  software  were  assumed  to  be  subject  to  wearout  with¬ 
in  the  interval.  But  this  is  not  generally  the  case  with 
software  systems.  Another  situation  where  such  an  assump¬ 
tion  might  be  justifiable  is  where  testing  intensity  increases 
within  the  interval  in  the  same  fasion  as  does  the  failure 
rate . 

This  assumption  is  used  in  models  TBF2 ,  TBF3 ,  and  FC10. 
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6.1.7  Testing  is  Representative  of  the  Operational  Usage 

The  test  cases  are  generally  chosen  to  ensure  that  the 
functional  requirements  of  the  system  are  correctly  met.  A 
given  user  of  the  system,  however,  may  not  use  the  functions 
in  the  same  proportion.  Testing,  then,  will  not  reflect  the 
operational  usage.  However,  information  about  the  usage 
profile  is  not  easy  to  obtain  and  this  assumption  would  be 
the  logical  one  to  use.  If  the  required  information  is 
available,  testing  effort  can  be  modified  to  be  representa¬ 
tive  of  the  use  profile. 

This  assumption  is  necessary  when  reliability  estimate 
based  on  testing  is  projected  into  the  operational  phase. 

It  is  used  primarily  in  models  IDB1,  IDB2,  and  IDB3.  Most 
TBF  and  FC  models  would  also  need  this  assumption  if  they 
are  being  used  to  assess  operational  reliability. 
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This  assumption  implies  that  all  remaining  faults 
are  equally  likely  to  appear  during  the  operational  usage 
of  the  system.  If  the  usage  is  uniform,  then  this  is  a 
reasonable  assumption.  If,  however,  some  portions  are  more 
likely  to  be  executed  than  others,  the  reliability  of  the 
system  can  be  recomputed  by  incorporating  this  information. 

In  other  words,  a  reliability  measure  conditioned  on  usage 
rather  than  an  unconditional  measure  would  be  more  desirable. 
If,  however,  such  information  is  not  available,  then  the 
assumption  of  uniform  usage  is  the  only  reliable  one. 

This  assumption  is  made  when  reliability  estimates  from 
any  model  are  based  on  the  number  of  remaining  faults. 
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6.1.9  Use  of  Time  as  a  Basis  for  Failure  Rate 


Most  models  use  time  as  a  basis  for  determining 
changes  in  failure  rate.  This  usage  assumes  that  testing 
effort  is  proportional  to  either  calendar  time  or  execu¬ 
tion  time.  Also,  time  is  generally  easy  to  measure  and 
most  testing  records  are  kept  in  terms  of  time.  Another 
argument  in  favor  of  this  measure  is  that  time  tends  to 
smooth  out  differences  in  test  effort. 

If  testing  is  really  not  proportional  to  time,  the 
models  are  equally  valid  for  any  other  relevant  measure. 
Some  examples  of  such  measures  are  lines  of  code  tested, 
number  of  functions  tested,  and  number  of  test  cases  exe¬ 
cuted  . 


6 . 2  Applicability  of  Existing  Software 

Reliability  Models 

Due  to  the  various  assumptions  that  are  required  by  the 
models  described  earlier  (see  Table  6.1),  it  is  necessary  to 
use  caution  in  choosing  a  model  for  software  reliability 
assessment.  In  this  subsection  we  develop  a  classification 
scheme  and  suggest  the  classes  of  models  that  might  be  appli¬ 
cable  in  various  phases  of  the  software  development  process. 

For  this  purpose  we  consider  the  following  phases  of 
the  software  development  process. 

1.  Design 

2.  Unit  Testing/Debugging 

3.  Integration  Testing/Debugging 

4.  Acceptance  Testing/Debugging 

5.  Operational  Testing 

6.2.1  Design  phase 

During  the  design  phase,  faults  (i.e.  design  faults) 
may  be  corrected  visually  or  by  other  formal/informal  pro¬ 
cedures.  Existing  software  reliability  models  are  not 
applicable  at  this  stage  because 

(i)  test  cases  needed  to  expose  faults  as  required 
by  fault  seeding  and  input  domain  based  models 
do  not  exist, 
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Table  6.1 


List  of  Key  Assumptions  by  Model  Category 


1.  TBF  Models 

.  Independent  times  between  failures 

.  Equal  probability  of  the  exposure  of  each  fault 

.  Embedded  faults  art  independent  of  each  other 

.  Immediate  fault  removal,  perfect  fault  removal, 
no  new  faults  introduced  during  correction. 

2 .  FC  Models 

.  Testing  intervals  are  independent  of  each  other 

.  Testing  during  intervals  is  reasonably  homogeneous 

.  Numbers  of  faults  detected  during  non-overlapping 
intervals  are  independent  of  each  other 

3 .  FS  Models 

.  Indigeneous  and  seeded  faults  have  equal  proba¬ 
bilities  of  being  detected. 

4.  IDB  Models 


Input  profile  distribution  is  known 
Random  testing  is  used 

Tnput  domain  can  be  partitioned  into  equivalent 
classes. 
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(ii)  failure  history  required  by  time  dependent 


models  is  not  available. 

6.2.2  Unit  testing 

The  typical  environment  during  the  module  coding  and 
unit  testing  phase  is  such  that: 

(i)  test  cases  generated  from  the  module's  input 

domain  may  not  form  a  representative  sample  of 
the  operational  "profile"  distribution, 

C i i )  times  between  exposures  of  module  faults  are  not 
random  since  the  test  strategy  employed  may  not 
be  random  testing.  Also,  test  cases  are  usually 
executed  in  a  deterministic  fashion, 

(iii)  exposed  faults  are  corrected  (debugged). 

Given  these  conditions,  it  seems  that  the  fault  seeding 
models  are  applicable  provided  it  can  be  assumed  that  the 
indigenous  and  seeded  errors  have  equal  probabilities  of 
being  detected.  However,  a  difficulty  arises  because  the 
programmer  is  also  the  tester  in  this  phase.  The  input 
domain  based  models  seem  to  be  applicable,  except  that 
matching  the  test  profile  distribution  with  the  operational 
profile  distribution  is  non-trivial.  Due  to  these  difficul¬ 
ties,  such  models,  though  applicable,  may  not  be  usable. 

The  time  dependent  models  (TBF  models)  do  not 
seem  to  be  applicable  in  this  environment  since  the  indepen¬ 
dent  times  between  failures  assumption  is  seriously  violated. 
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6.2.3  Integration  testing 

A  typical  environment  during  integration  testing  is: 

(i)  modules  are  integrated  into  partial  or  whole 

systems  and  test  cases  are  generated  to  verify 
the  correctness  of  the  integrated  system, 

(ii)  test  cases  may  be  generated  randomly  following 

an  input  distribution  or  may  be  generated  deter¬ 
ministically  using  a  reliable  test  strategy,  the 
latter  being  probably  more  effective, 

(iii)  exposed  faults  are  corrected  and  there  is  a  strong 
possibility  that  the  removal  of  exposed  faults 
may  introduce  new  faults. 

Fault  seeding  models  are  still  theoretically  applicable 
since  we  still  have  the  luxury  of  seeding  faults  into  the 
system.  Input  domain  based  models  based  on  an  explicit  test 
profile  distribution  are  also  applicable.  Again,  the  diffi¬ 
culty  in  applying  them  at  this  point  is  the  large  number  of 
logic  paths  generated  by  the  whole  system. 

If  deterministic  testing  (i.e.,  boundary  value  analysis, 
path  testing,  etc.)  is  used,  times  between  failures  (TBF) 
models  may  not  be  appropriate  because  of  the  violation  of 
the  independence  of  inter-failure  times  assumption.  Deter¬ 
ministic  testing  increases  the  likelihood  of  exposing  faults 
and,  hence,  inter- failure  times  are  no  longer  random.  Failure 
count  models  (ex.  FC1 ,  FC2)  may  be  applicable  if  sets  of 
test  cases  are  independent  of  each  other,  even  if  the  tests 
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within  a  set  are  run  deterministically.  This  is  so  because 
in  FC1  and  FC2  models  the  system  failure  rate  is  assumed 
to  decrease  as  a  result  of  executing  a  set  of  test  cases  and 
not  at  every  failure. 

If  random  testing  is  done  according  to  an  assumed  input 
profile  distribution,  then  most  of  the  existing  software 
reliability  models  are  applicable.  Input  domain  based  models, 
if  used,  should  utilize  a  test  profile  distribution  which  is 
statistically  equivalent  to  the  operational  profile  distri¬ 
bution.  Fault  seeding  models  are  applicable  likewise,  since 
faults  can  be  seeded  and  the  equal  probability  of  fault 
detection  assumption  may  not  be  seriously  violated.  This  is 
due  to  the  random  nature  of  the  test  generation  process. 

Times  between  failures  and  failure  count  models  are  most 
applicable  in  this  environment  with  random  testing.  The 
only  question  in  applying  these  models  is  which  specific  model 
to  use.  Some  people  prefer  to  try  a  couple  of  these  models 
for  the  same  failure  history  and  then  choose  an  appropriate 
one.  However,  because  of  the  different  underlying  assump¬ 
tions  of  these  models,  there  is  still  a  subtle  distinction 
as  to  when  a  specific  model  is  most  applicable.  For  example, 
for  operating  systems  or  real-time  systems  which  are  run 
almost  continuously,  the  time-dependence  assumption,  say  of 
the  Non-homogeneous  Poisson  Process  Model  (Fcl),  or  model 
FC2,  are  most  applicable.  We  should,  however,  be  aware 
of  the  fact  that  in  most  real-time  systems  the  inputs  to 
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the  software  may  not  be  random. 

If  there  is  reason  to  believe  that  the  operational 
input  to  the  software  is  essentially  uniform,  then  each 
embedded  fault  in  the  software  has  essentially  an  equal 
chance  of  being  detected.  Thus,  times  between  failures 
models  based  on  a  constant  multiple  of  the  number  of  re¬ 
maining  faults,  (e.g.  Model  TBF1  of  Jelinski  and  Moranda) 
are  more  applicable.  In  real-world  situations,  the  equal 
chance  of  fault  detection  assumption  is  rarely  true.  This 
is  due  to  two  main  reasons: 

(i)  embedded  faults  in  the  software  have  unequal 
sized,  some  are  large  while  others  are  small, 

(ii)  the  input  distribution  may  not  be  uniform. 

If  such  is  the  case,  models  which  assume  that  error 
occurrence  rate  varies  with  time  (e.g.  Model  FC1)  are  more 
applicable: 

In  an  environment  where  exposed  errors  are  imperfectly 
debugged,  a  theoretically  more  applicable  model  is  the 
Imperfect  Debugging  Model  (Model  TBF6 )  . 

In  environments  where  imperfect  debugging  is  assumed 
and  additional  errors  are  introduced  during  the  error  correo 
tion  process,  none  of  the  existing  software  reliability 
models  is  applicable.  Models  of  this  kind  are  yet  to  be 
developed. 
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During  acceptance  testing  the  software  is  given  to 


"friendly  users".  These  users  generate  inputs  (usually 
random  following  an  input  distribution)  to  verify  the  cor¬ 
rectness  of  the  software.  The  main  differences  between  this 
environment  and  that  of  the  integration  testing  environment 
are  the  following: 

(i)  exposed  faults  are  immediately  corrected  since 
the  user  may  not  know  how  to  correct  them, 

(ii)  the  user  does  not  have  the  luxury  of  seeding 
faults  in  the  program, 

(iii)  the  user  may  not  even  know  the  structural  paths 
of  the  program. 

It  can  be  easily  seen  that  the  fault-seeding  models  are  not 
applicable  because  of  (ii)  above.  Times  between  failures 
models  are  also  not  applicable  since  these  models  assume 
correction  of  exposed  errors  before  testing  proceeds. 

6.2.5  Operational  phase 

When  the  reliability  of  the  software  as  perceived  by 
the  developer  or  the  "friendly  users"  is  already  acceptable, 
the  software  is  released  for  operational  use.  During  the 
operational  phase,  the  user  inputs  may  not  be  random.  This 
is  because  the  user  may  use  the  same  software  function  or 
path  on  a  routine  (production)  basis.  Inputs  may  also  be 
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correlated  (like  in  real-tine  systems),  thus  losing  their 
randomness.  Furthermore,  faults  are  not  always  immediately 
corrected . 

Software  that  is  in  operational  use  may  be  recalled 
by  the  developer  so  that  errors  exposed  during  the  opera¬ 
tional  phase  can  be  corrected.  The  usual  practice  is  to 
release  a  newer  version  of  the  software  which  hopefully 
contains  none  of  the  exposed  faults. 

It  anpears  that  the  failure  count  models  could  be 
used  in  this  phase  if  the  input  process  can  be  considered 
to  be  random  over  longer  intervals  of  time. 
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7.  STEP  BY  STEP  PROCEDURE  FOR  SOFTWARE 
RELIABILITY  MODELING  AND  ILLUSTRATIVE 
EXAMPLES 

The  purpose  of  this  section  is  to  describe  a  systematic 
procedure  for  developing  a  software  reliability  model  from 
available  failure  data.  The  general  procedure  is  de¬ 
scribed  in  Section  7.1  and  illustrated  via  analyses  of 
some  failure  data  in  Section  7.2.  An  important  phase  in 
the  development  of  a  model  is  the  estimation  of  parameters, 
mostly  done  iteratively.  Details  of  such  calculations  for 
the  data  set  of  Section  7.2  are  given  in  Section  7.3. 
Estimation  of  parameters  is  further  illustrated  for  a 
simple  data  set  in  Section  7.4  using  the  De-Eutrophication 
model  of  Jelinski  and  Moranda  (Model  TBF1)  and  the  NHPP 
model  of  Goel  and  Okumoto  (model  FC1) . 
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The  step  by  step  procedure  is  shown  in  Figure  7.1 
and  described  below. 

Step  1 ;  Study  the  failure  data 

Most  of  the  models  discussed  in  this  report  require  the 
prior  existence  of  some  failure  data  to  fit  a  model.  The 
first  step  in  developing  a  model  is  to  study  this  data  in 
order  to  gain  an  insight  into  the  nature  of  the  process 
being  modelled.  If,  for  example,  the  number  of  failures  per 
unit  time  is  increasing,  it  would  appear  that  more  and  more 
functions  are  being  added  during  the  subsystem  or  system 
test.  The  data  may  have  to  be  normalized  before  proceeding 
any  further  because  most  of  the  models  assume  a  basically 
steady  system.  It  may  be  that  the  test  case  severity  is 
increasing.  Again,  this  needs  to  be  accounted  for  in  the 
modelling  process. 

Step  2:  Choose  a  Reliability  Model 

The  next  step  is  to  choose  an  appropriate  model  based 
upon  an  understanding  of  the  testing  process  and  the  assump¬ 
tions  of  the  various  models  discussed  earlier.  The  data  in 
Step  1  can  also  be  used  to  sharpen  decisions  about  the  applica¬ 
bility  of  a  class  of  models  or  of  the  choice  of  a  given  model. 
Step  3:  Obtain  Estimates  of  Parameters  of  the  Model 


Different  methods  are  generally  required  depending 
upon  the  type  of  available  data.  The  most  commonly  used 


Figure  7.1  Flowchart  for  Software  Failure  Data 
Analysis  and  Decision  Making. 


ones  are  the  least  squares  and  maximum  likelihood 
methods . 

Step  4:  Obtain  the  fitted  model. 

The  fitted  model  is  obtained  by  substituting  the 
estimated  values  of  the  parameters  in  the  chosen  model. 

At  this  stage,  we  have  a  fitted  model  based  on  the 
available  failure  data. 

Step  5:  Perform  goodness  -  of - fi t  test. 

Before  nroceeding  further,  it  is  advisable  to  conduct 
the  Kolmogorov-Smirnov  goodnes s - of - f i t  test  or  some  other 
suitable  test  to  check  the  model  fit. 

If  the  model  fits,  we  can  move  ahead.  However,  if 
the  model  does  not  fit,  we  have  to  collect  additional 
data  or  seek  a  better ,  more  appropriate  model.  There  is  no 
easy  answer  to  either  how  much  data  to  collect  or  how  to 
look  for  a  better  model.  Decisions  on  these  issues  are 
very  much  problem  dependent. 

Step  6:  Compute  confidence  regions. 

It  is  generally  desirable  to  obtain  801,  90%,  95%  and 
99%  joint  confidence  regions  for  the  parameters  of  the  model 
to  assess  the  uncertainty  associated  with  their  estimation. 
This  information  is  helpful  in  the  interpretation  of  model 
outputs . 
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At  this  stage  we  can  compute  various  quantitative 


measures  to  assess  the  performance  of  the  software  system. 
Some  useful  measures  are  shown  in  Figure  7,1.  Confidence 
bounds  can  also  be  obtained  for  these  measures  to  evaluate 
the  degree  of  uncertainty  in  the  computed  values. 

Step  8:  Decision  making. 

The  ultimate  objective  of  developing  a  model  is  to 
use  it  for  making  some  decisions  about  the  software  system, 
e.g.,  whether  to  release  the  system  or  continue  testing. 
Such  decisions  are  made  in  this  step  of  the  modellj..  >7 
process  based  on  the  information  developed  in  the  previous 


7-2  An  Example  of  Software  Reliability  Modelling 

In  this  section  we  employ  the  above  step-by-step 
procedure  to  illustrate  the  development  of  a  software 
reliability  model  based  on  actual  failure  data  from  a 
real-time  command  and  control  system.  For  the  purposes  of 
this  illustration,  we  emplov  the  NHPP  model  of  Goel  and 
Okumoto  (Model  FC1). 

The  delivered  number  of  object  instructions  for 
the  system  being  modeled  was  21,700  and  the  system  was 
developed  by  Bell  Laboratories  [M.US80]. 

Step  1: 

The  original  data  was  given  as  times  between  failures. 

To  overcome  the  independence  assumption,  we  summarized 
the  data  into  numbers  of  failures  ncr  hour  of  execution 
time.  The  data  are  shown  in  Table  7.1  and  plotted  in  Fig.  7.2. 
A  plot  of  N(t) ,  the  cumulative  number  of  failures  is  shown  in 
F'-  7.3  with  other  quantities  to  be  discussed  later. 

A  study  of  the  plot  indicates  that  the  failure  rate 
(number  of  failures  per  hour)  is,  in  fact,  decreasing  and 
hence  NHDP  (Model  FC1)  can  be  employed  to  model  the  failure 
process . 

Using  the  method  of  maximum  likelihood  (See  Appendix 
D-l),  estimates  of  the  parameter'  a  and  h  were  obtained 


and  are 
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Fig.  7.3.  Number  of  Failures  and  90%  Confidence  Bounds  (SYS1) 


A 

a  =  142.32  (Expected  number  of  faults) 
b  =  0.1246  (Faults  per  fault  per  hour) 

For  details  of  these  computations,  see  Section  7.3. 

Step  3 

The  model  based  on  the  data  of  Table  7.1  is 
m(t)  =  142.32(1  -  e'0.1246tJ 

and 

A ( t)  =  17.73  .eT°-I246t 

A  plot  of  m(t)  versus  t  is  shown  in  Figure  7.3 
Step  4. 

We  used  the  Kolmogorov- Smirnov  goodness  of  fit  test 
for  checking  the  adequacy  of  the  model.  For  details  of 
this  test  see  Goel  [GOE82].  Details  of  the  test  for  this 
data  set  are  shown  in  Table  7.2.  We  note  that  on  the  basis 
of  this  test  the  model  seems  to  indicate  a  good  fit. 

Step  5. 

The  confidence  regions  for  a  and  b  were  computed  but 
are  not  shown  here. 

Step  6. 

ve  computed  three  performance  measures:  the  cumula¬ 
tive  number  of  failures,  the  expected  number  of  remaining 
faults  and  software  reliability.  These  measures  are  plotted 
in  Figures  7.3,  7,4,  and  7,5,  respectively. 

Plots  of  the  confidence  bounds  for  the  expected  number 
of  failures,  expected  number  of  remaining  faults  and  relia¬ 
bility  are  also  shown  in  the  above  figures. 
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Table  7.2 


Kolmogorov-Smirnof £  Test  for  Data  Set  of  Table  7.1 


*1 

H(t.) 

W 

lGo(ti>  -  H(*i>l 

IGq^)  -  H(trl)| 

1 

.198 

.123 

.076 

.123 

2 

.316 

.231 

.085 

.032 

3 

.397 

.326 

.071 

.010 

4 

.470 

.410 

.060 

.014 

5 

.551 

.485 

.066 

.014 

6 

.602 

.551 

.051 

.000 

7 

.617 

.609 

.008 

.006 

8 

.654 

.660 

.006 

.043 

9 

.676 

.705 

.029 

.051 

10 

.683 

.745 

.061 

.068 

11 

.713 

.780 

.067 

.096 

12 

.764 

.811 

.047 

.098 

13 

.779 

.839 

.059 

.074 

14 

.816 

.863 

.047 

.084 

15 

.852 

.885 

.032 

.068 

16 

.897 

.903 

.006 

.050 

17 

.897 

.920 

.023 

.023 

18 

.933 

.935 

.001 

.038 

19 

.941 

<948 

.007 

.014 

20 

.948 

.959 

.011 

.018 

21 

.963 

.969 

.006 

.021 

22 

.970 

.978 

.008 

.015 

23 

.985 

.986 

.001 

.016 

24 

.992 

.993 

.001 

.008 

25 

1 

1 

0 

.007 

D25,  .20  «  .208>  Dmax 
025,  .05  *  .264 >0^ 


0.123 


> 


0.123 J 


Hence  the  model  fits  the  data 
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Fig.  7.4.  Observed  and  Expected  No.  of  Remaining  Faults 
with  901  Confidence  Bounds  on  IE 
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X  study  of  these  plots  indicates  that  the  NHPP 
model  (Model  FC1)  provides  an  excellent  fit  to  the  data 
and  can  be  used  for  purposes  of  describing  the  .failure 
behavior  as  well  as  prediction  of  future  failure  process. 

The  information  available  from  this  can  be  used  for  planning, 
scheduling,  and  other  management  decisions. 

Step  8 

A  plot  of  the  reliability  growth  is  shown  in  Figure 
7.6.  To  obtain  this  plot,  we  recomputed  the  parameters  a 
and  b  at  one  hour  intervals  based  on  data  from  15,  16,  .,,,  25 
hours  of  execution  time  testing.  The  one  hour  ahead  pre¬ 
dicted  reliability  at  each  of  these  points  is  what  is  shown 
in  Figure  7.6. 
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S/W  RELIABI 


Fig.  7,6  S/W  Reliability  Growth  with 
Parameter  Updating 


7.3  Details  of  Parameter  Estimation  for  the 


Data  of  Section  7.2. 


The  data  in  Section  7.2  can  be  written  as  follows. 


Times 


t. ( x  3600  secs) 

1 

2 

3 

4 


Cumulative  #  of  Failures 
(yi,  i =  1,2,  . 25) 

27 

43 

54 

64 


24 

25 


135 

136 


Estimation  of  Parameters  a  and  b 

MLE  of  a  and  b  can  be  obtained  by  solving  the  following 
pair  of  equations 

'btn 

bt-s  -bti.. 


a(l  -  e  )  =  y 


bt  n  (y.  -  y.  JCt.e  1  -  t.  .e 

n_  r  w  l  7 1- 1' v  1  l-l  1 


ate  =  l 
n  iii 


e  -  e 


The  above  two  equations  yield 

-bt  -bt. 

ynV  ,  ?  (yi  -  yi-l)(tie  iVl 

:^t_  .  -bt  .  .  -bt  • 

(1  -  e  n)  11  e  1-1  -  e  1 


e  bt*-b 


MLE  of  b  can  be  obtained  from  this  using  Newton-Raphson  method. 
Then  we  have 

2  ^n _ 

-6t 

(1  -  e  n) 
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Use  of  the  Newton-Raphson  method  to  find  b 


Let 


_  ?  ‘-1-  ’e 


F  i  I 

i  =  l 


-bt.  -bt; 

i-i1 


">  Vne 


-b<l-l  -bti 
e  -  e 


-bt, 

rFt" 


1  -  e 


n 


then 


-bt.  i  -bt^  2  2 

dF  _  n  hyiWji!  'e  Hti-le  -‘i1 

m  'i-1  (e'bti-l.e'bti)2 


-btj 
(tie  ‘-tj.j 


*  b  t  •  i  ^  r  b  t 

e  ‘11, 

(l-e'bS2 


For  the  above  data  set 


n  *  25,  tQ  a  0,  t^  =  i  for  i=l,2,  ....  25 

y0  ■  o,  y:  ■  27  >  y2  s  43»  •••»  >^4  =  135,  y25 


=  136. 


Iteration  1 


Let  initial  b  *  b^  =  0.01  then 


'  i2!  ‘-““i-l  /'“i 


-  .Olt.  -.Olt.  ,  7C 

"i-i6  __}  - 

(1  -  e'0,25) 


12665.11333 


-  e 

136  x  2 5e' *  25 

- — r~ — 

(1  -  e  -Zb) 


12665.11333  -  11970.75966  =  694.35367 


-.Olt.  ,  -.Olt. 

dF  2r5  <>Vyi-i^e  *e  ><* i-1 e 

3F  "  J, 

11  -  m  ♦  -.Olt 


-.Olt-  ,  7  -.Olt- 

1  -tje  X) 


-.Olt.  -.Olt.,  2  2 

-(t,e  X)  ]  y25t25e 


25 


l _ l-l  _ 

-.olt.  .  - . Olt ."  ” 

Ce  1-1  -e  V 


* • 01t2j  2 

(1-e 


-1359988.667  +  1352938.747 
-7049.920 
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Ab(1)  -  -F  t  ||  =  -694.35367  *  (-7049.920) 


0.0984910087  >  0.0001  =>  Go  to  Iteration  2 


Iteration  2 


b(2)  ,  bCD  +  AbCl)  a  ,Q1  +  0.0984910087 

=  ,1084910087  *  .108 


- . 108t •  - . 108t - 

P  V  (yi'yi-l)(tie 
*  =  -.lOflt/'j  - , l08i:i 

1-1  e  -  e 


136  x  25e 
1  -  e'  * 


- . 108x25 


*  319.7894262  -  241,7604258  =  78.029000 


dF  .  “  'y 

3F  ,1,  ~ 


- . 108t .  ,  - .  108t .  - 


- . 108t ■  ,  ,  - . 108t . 

>  1'1-t:e  x) 


-(tie 


- . 108t .  - . 108t .  .  - 


‘Vl6 


)  ]  .  136  x  252 x e' • 108x25 


-  .  108t  -  .  -.11 

(e  l’-e 

=  -11543.18018  +  6473.77611 

-  -5069.40407 


l  2 

V 


(1  -  e'* 


,(2)  s  -p  i  S  *  -78.0290004  *  (-5069.40407) 


=  .0153921446  >  .0001  = 
=> Go  to  Iteration  3 


Iteration  3 


b(3)  _  b(2)  +  Ab(2)  =  ,1084910087  +  .0153921446 


=  .1230831533  .124 


- . 12  4t. 


25  (yi-yi.^Che^-w 


- . 124t . 


-. 124t. 


136  x  25e 
(1  -  e'* 


124x25 


164.2123219  -  160.8842808  =  3.3280411 
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-.1241,,  -.124t,  ,  -.1244,,  2  -.1244, 

25  Oyr^)  (a  1  1)(4^e1  1-4?.  x) 

i  =  1  -  .  124t  -  -  .  124t .  -.  7 

'(tie  ’ ti- le  l "  A  136  x  252  x e~‘ 124x25 


V 


(1  *  e‘* 


-8850.321535  ♦  4212.428725 
-4637. 892810 


-F  f  ||  =  -3.3280411  t  (-4637. S92810) 

*  7.175761232 x  10*  >  0.0001  => Go  to  Iteration  4 


Iteration  4 


b(4)  =.  b(3)  ♦  Ab(3^  -  .1238831533  +  .0007175761232 


=  .1246007294  =  .125 
-  .  12St .  -  .  1 2 5 1 

r  2,5  ^r^-i^v  '-Vi6 

F  l.  - -TirstT  " — -.irsr 

lml  (e  ^-e  x) 


1  -.125x25 

136  x  25  x  e 

- - -  -ATSxJT— 


157.8981635  -  157.8910124  =  0.0071511 


dF  2rs  fri-yi-i5  (e 


. 1 2  5 1 -  - . 12  5t  -  , 

±  L  „  M  f  f  “ 


)(t:_ie 


(1  -  e 


1 2  5 1  i  _  i  -  •  1 2  5 1 , 


- . 125t . 

-CV  •ti.1e 


-  .  12  5 1  -  ,  , 

1  -t^e  1'1)-  156  x  252  xe--125x25 

‘i-1  - (1.,-1^)T- 

-  e  ) 


=  -8748. 547027  +  4130.580986 
=  -4617.966041 

,(4)  ,  -p  •  3F  =  - .0071511  *  (-46.7.966041) 

=  1. 5485388 x  10'6  <  0.0001  * 

b  *  b^  +  Ab^4-1  -  .  1246007294  +  .0000015485 

=■  .  12460227"9  -  .  1246 


(1  -  e'~bt25)  (1  '  e'bx2S)  U  '  e'-l246x25) 


142.3156405 
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7 . 4  Numerical  Examples 


Once  a  model  has  been  identified,  the  key  step  in 
fitting  the  model  to  the  observed  failure  data  is  the  esti¬ 
mation  of  parameters.  This  requires  detailed  computations 
because  in  almost  all  cases  the  likelihood  equations  have 
to  be  solved  iteratively  using  numerical  techniques. 

In  this  section  we  show  the  details  of  such  compu¬ 
tations  for  selected  data  sets.  Two  types  of  data  are 
considered  viz,  times  between  failures  and  failure  counts. 
Computations  for  some  performance  measures  are  also  given. 
The  two  models  considered  are  the  Jelinski-Moranda 
De- eutrophication  Model  (Model  TBF1)  and  the  Goel  Okumoto 
NHPP  model  (Model  FC1) . 

The  data  set  consists  of  only  a  few  points  so  that 
the  computations  could  be  repeated  using  a  hand  calculator. 
It  should  be  pointed  out  that  the  leading  decimal  places 
have  been  retained  in  these  examples  to  ensure  that  the 
round-off  errors  do  not  build  up. 


7.4.1  Example  showing  computations  for  the  Jelinski-Moranda 
Model  (Model  TBF1) 

Consider  a  set  of  failure  times  as  follows: 


Failure  no. 


Time  Between  Failures  t. 


The  parameters  to  be  estimated  are  N  and  <{> 

The  maximum  likelihood  estimate  of  N  can  be  obtained  from 
the  following  equation  using  Newton-Raphson  method: 


n  1 

Ji 


l  t. 


We  can  then  find  $  from  the  following  equation  by  substituting 


N  for  N 


N(  l  t.)  -  l  (i-l)t. 
i=l  1  i=l  1 


Use  of  Newton-Raphson  method  to  find  N 


_  ?  1  . 

~  A,  N  -  ( i— 1) 


-  -TT^—  U. 

,L  11 


then 


dF  - 
3n  ~ 


-  -5-i—  (.1  (1-1)  t  ))' 

t  1“1 


J  1 _ ^ 

i-l(N  -  (i-1)  >‘ 


l  t± 

i=l  1 
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n  5 

For  the  above  dataset  n=5,  £  t.=  £  t.  *= 

i=l  1  i=l  1 


-  (3  +  30  +  113  +  31  +  115)  *  292?  and 

n  5 

l  (i-l)t.  =  l  (i-l)t.  =  t-  +  2t,  +  3t .  +  4tc 

i-1  1  i=l  i  ^  i  4  5 

-  30  +  226  +  93  +  460  =  809 


Iteration  1:  Let  an  initial  value  of  N  be  N 

5 


(1) 


=  5 


Then  F 


(1)  _  r  1 

'  ill 


5  - 


*  (809) 


_  ,1  .  1  ^  1  ^  1  .  ,,  5 

-  <5+?  +  i  +  ?+l>  -  2722945'2055 


and 


=  2.283333333  -  2.242703532 

,-2 


=  4.062980  x  10 


dF 


(1) 


dN 


HT 


(5  - 


-  Z 


797 


(809)  r  i-1  {5  -  i  +  1}‘ 


{2.229452055} 
,  Cl) 


7  ~  (2?  +  TS  +  I  +  k  +  1} 


or 


dF 


dN 


nr 


=  -4.57667284  X  10 


-1 


Next 


AN(1)  =  -  F(1)  t  dF|^y  =  (4.062900 x 10"2)  *  ( 4 . 57667284 x  10_1) 

dN 

=  0.08877584602 


-4  (1)  -4 

Let  error  €£10  .  Since  AN v  10  ,  we  go  through  the  next 


iteration 


Iteration  2.  The  value  of  N  for  the  second  iteration  is 
n<2)  a  N(l)  +  -  5.08877584602 


F<2>  =  | 

i=l 


5.68877$74'602"-~i  +  l' 


5.08877584602  - 


W 
75 7 


and 

dF(2) 

dN*2> 


or  F(2)=  5.22788908141  x  10-3 

5 

-  I 


{5.08877584602  -  §|f  }  2  i=1  (5-08877584602  -  i +  l)2 


Now 


AN 


=  .93037439 04  -  1.276022534  =  -3.45648143667  x  10 


t-j\  (j)  je<  ( 2 )  - 

=  =  (5.22788908141  x  10~JH  ( 3 . 45648143667  x  10_1) 


-1 


dN 


TO 


■  0.0151248869 
.-4 


Since  this  number  is  >10  ,  we  go  to  the  next  iteration. 

Iteration  3.  Value  of  N  for  the  third  iteration  is 


Nt3)  *  tl(2)  +  AN(2)  =  5.08877584602  +  0.0151248869 

=  5.10390073293 


„(3)  _  r  1 

F  *  A  5.10396073253  -  i  +1 


i=l 


5.10390073293  - 


■777 

757 


2.142960589  -  2.142839277  -  1.21312176929  x  10 


-4 


and 


dF 


(3) 


dN 


AN 


TO 


(3) 


5  Vi 

- -  l  - »r 

(5.10390073293  -  i=«l{5 . 10390073293  -  i  +  1 } 


.9183520332  -  1.248093532  =  -3.29741499203  x  10 


-1 


-F 


(3)  .  dF 


(3) 


dN 


TO 


(1.21312176929 x 10-4)  *  (3 . 29741499203 x  10_1) 


3.679008472  x  10 


-4 
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Since  AN 


(2)  -4 

>10  ,  we  continue  the  computations 

Iteration  4.  The  value  of  N  for  the  iteration 

N(4)  =  N(3)  +  AN(3)  =  5.10390073293 + 3.679008472 x 10-4 

=  5.10426863377 


(4) 


y  _ 1  . 

i£1  5.10426863377  -  i  +  1 


5.10426863377  - 
*  2.142501537  -  2.142501468  =  6.93541917229  x  10 


797 
-8 


dF 


(4) 


dN 


T4T 


{5.10426863377 


_ _  _  f  _ 1 _ 

-  i=l  (5.10426863377  -  i  +  l)^ 


=  0.9180625077  -  1.247427058  =  -3.29364549966  x  10 


-1 


and 


an 


(4)  m 


-F 


(4)  i  dF 
dN 


(4) 

m 


(6.9354191722 x  10'8)  *  (3 . 29364549966  x  10_1) 
2.105696916  x  10-7 


Since  AN ^  <io  ’  we  terminate  further  iterations  and  compute 
N  as 


or 


N  =  N(41  + 


AN 


14} 


=  5.10426863377  +  2.105696916  x  10 


-7 


N  *  5.104268844 
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Using  the  above  value  of  N »  we  get 


For  this  data  set,  we  get 

N  =  5.104268844 
$  =  7.337333131  x  10 


Performance  Measures 

After  estimating  the  parameters  N  and  <J>  ,  we  calculate 
the  performance  measures  using  these  estimates  as  follows: 
•Reliability  at  time  t  after  nth  failure  Rn(t) 


''.4.2 


Examples  showing  commutations  for  the  Goel-Okumoto 
Model  (Model  FC11 

Consider  the  following  failure  counts  based  on  the  data 


of  the  previous  example. 

Interval  No. 


1 

2 

3 


Cumulative  No.  of 
failures 

2 

4 

5 


The  parameters  to  be  estimated  are  a  and  b- 

MLE  of  a  and  b  can  be  obtained  by  solving  the  following 

pair  of  equations 
-bt  . 

a  ( 1-e  n)  =  y  . 

.  .-bt,  _  ;  !tie'  i-1) 

at  e  n  *  l  — .  -  -  . . .  ■  . . 

i-1  -bt.  .  -bt. 

e  i-1-e  i 

These  equations  yield 

yntne”btn  "  (yi"yi-l}  ^tie~bti“ti_1e‘bti-l) 

=  r  -bt."  , - ~bt“ - 

0  X-l —0  ^ 


(l-e'btn)  i-1 


which  can  be  used  to  obtain  the  mle  of  b  by  the  Newton-Raphson 
method.  Then  we  can  compute  the  mle  of  a  as 


:i-en) 


Use- of  Newton-Raphson  method  to  find  b 

n  (Yi'Y.1  J  (t.e’^i-t.  7e“bti-l)  y  t  e~btn 

Let  F  =  Z  1-1  J- - - - - 

i-i  -bt.  ,  -bt .  ,,  -bt  . 

1_l  e  i-l_e  i  d-e  n) 


then 


dF 

db 


-bt,  ,  -bt.  ,  -bt. .  -bt. 
n  (yi'yi-l)l(e  1_e  1)  (t-_i«  1-1  -t*e  x) 

i=l 


-bt . 


(e 


^  -  7*7 


-bt,  -bt.  ,  2  2  — bt- 

■V8  -  tj.s  Vi 

-bt,  ,  -bt.  i  ^Ft  S’ 


(e 


■i-1 


-  e 


•) 


(1-e 


n 


For  the  above  data  set 


n  «  3f  tQ  -  0,  t2  -  1,  t2  «  2,  t3  -  3 

y0  =  o,  v1  =  2,  y2  =  4,  y3  =  s 

Iteration  1.  Let  an  initial  value  of  b=.01,i.e.,  b^=0.01 


Then 

-.Olt. 

„  ?  'yi-n-i1  (V  -V;' 

■  ill — 


-.Olt 


i-1 


-.Olt- 


)  y3t3e 


-.Olt.,  -.Olt. 
i  1-1  -  e 


-.oit: 


-.Olt, 


yl(tle 


(1  -  e 

-.01t2  -.Olt, 


-:m" 


)  (y2-yi)(t2e 

~  +  -70  IF,  -.olt 


-tle 


) 


1  -  e 


-  e 


-.Olt. 


-.Olt. 


-.Olt- 


(y3  -  y2}  (t3e 

-.olt'  -.bit 


-fc2e 


)  y3t3e 


-.Olt. 


-  e 


( 1  -  e 


2<e~-01)  .  (4-2)(2e_— ^e 

1  I  e"~  •  0"I e-^  -  e"^ 

.  5*  3*e~*03 
(1  -  e-"*  51) 


-.°2---.01>  (5  -  4?  (3e~ * 03  -  2e~ 


-.02  -'.0T 

e  -  e 


493.5041667  -  492.5374994 
.966667222210 


Now, 

dF  _3  [(e 

® 


’•01ti-l  ■'01llw  2  ■•01ti-l 

-  e  )  (t^^e  -  t 


-.Olt. 

-  (V  -Vie 


-.Olt, 


i_1)2]  y  t2e~*°lt3 

-.Olt.  .  -.Olt.  2  + 

(e  1-1 -e  *) 


-TflltT  r 

(1  -  e  J) 


y^e"*01*0  -  e-*01)  {-e-’01 )  -  e”,01x2] 


=  -49999.58334  +  49996.25017 


■  -3.33316667270 
and 

Ab(1)  =  -F t dF/db  =  -.966667222210 
=  .2900146669  >  .0001 
Let  €  =  0.0001.  Since  Ab(1)>€, 

Iteration  2.  =  b  ^  +  Ab  ^  = 


f  -3.33316667270 

we  go  to  iteration  2 
.3000146669  =  .3 


10.2906087  -  10.27600431 


0146665525129 


Now, 

-.3t.  ,  - . 3t .  ,  -.3t.  .  2  - . 3t . ) 

dF  =  3  (yj-yi-l)  [  (e  1  >  (tj_le  “  fcie 

315  i-1  -  •  3t .  ~  •  3t,  ,  -  ,  ~ .  3t, 


*  (tie 


-  Vl6 


-  .  3t  ■  i  •  -»  '-i  2 
(e  1  1  -  e  V 


i_1)2]  y3t2e 
-7Tt7“ -  +  - r7Tt 


(1  -  e 


3)2 


-  21(1-  e~  ‘  3 )  (~e~  ‘  3 )  -  e~‘6] 

— —y-j 

,  2[  (e'*3  -  e"‘6)  (e"-3  -  4e'*6)  -  (2e‘‘6  -  e-*3) 2J 

+ - — n — rnrr2 - 

(e  -  e  ) 

].[  (e-*6  -  e"‘9)  (4e"‘6  -  9e~*9)  -  (3e~*9  -  2e“  * 6  )  2  ]  .  45e"*9 

+  - - _  a  2 -  +  - r  a~  a 

(e  -  e  (1  -  e  ’  V 

,  -.3  ,  - .  9  -1.5  ,.-.9 

(“  -.2  2  7  - .'3 .6.  2  +  77=7?  7". 9,  2  5  +  7:  Z-.9.2 

(1-e  )  (e  -e  )  (e  -  e  )  (1-e  ) 

=  -55.13532562  +  51.94726586  =  -3 . 1880597558* 

Ab(2>  «  -F  t dF/db  =  -.0146665525129  *  -3.18805975584 

=  .004600463491 

( 2 ) 

Since  Ab  >€,  we  continue  with  iteration  3. 

Iteration  3  .  b(3)  =  b(2)  +  Ab<2)  =  .  3000146669  +  .004600463491 

=  .30461513036  t  .305 

-.305t3 

- .  305 1 .  :  -TSOStZ  *  -.3o5t, 


- . 305t .  - . 305t .  . 

.  r3  (yi-yi-l)(ti6  “Vl* 

F  ill  - -■“5‘w  - 


)  y3t3e 


-  e 


(1-e 


) 


2(e“*305)  t  2(2e“’610  -  e"*  305)  .  1  ( 3e" ’ 915  -  2e- * 6 10) 

: — +  — — -tsio-  +  — =7sts — =7Tn — 

l-e  e  ~ 

-.915 


-  e 


-  e 


5  •  3  •  e 


(1  -  e'-915) 

10.04088223  -  10.04087226  *  9.96719302759  x  10 


-6 
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I 

*  ■  *• 


- . 305t .  .  - . 305t .  -  - . 305 t 4  .  ,  - 

dF  3  tYcYt-Jie  -•  >^-ie  -*?• 

3E  =  .1,  - 

X-"  x 


-  (t.e 


- . 305t •  - . 305t.  ,  2  7  - . 305t, 

'  fci-ie  >  1  ^ 

3'65t7" "  —.30 B't .  2  +  =TJ75't' 


(e 


‘i-1 


-  e 


•) 


(1  - 


3)2 


21 (1  -  e"'305) <-e‘,  305l  -  e"'610] 

■  (i-.-34V - 

2|,e--305  -e-610)(e--305  -  4e-610)  -  (2e‘  ■ 610  -  e‘  • 305)  2] 

+  - J65 — '-.S16,'2 - 


(e 


-  e 


) 


.  lKe"*610  -  e“*915)  (4e~‘610  -  9e‘*915)  -  (3e~*915  -  2e“*610) 

+  - - _-.9iT;2 - 


(e 


-  e 


) 


...  «“.915 

45  -e 

+  - -  gr;  i 

(1  -  e 

-.305 


2e 


(1  -  e--  305)  2  (e 


.  -.915  -1.525 

2e  0 

+  .  3(55  7r75ToT2  +  ,  -.610  .-T915.  2,} 

-  e  )  (e  -  e  ) 


45e 


-.915 


(1  -  e-*915) 2 

■  -53.470157  +  50.28643996  =  -3.18371704092 
*b(3)  =  -F  *  dF/db  =  -9.96719302759  x  10"6  -  -3.  18371704092 

=  3.1306781 x  10*6 
Since  Ab(3) <e,  we  get 

b  =  b(3)  +  ib(3)  =  3.0461826104 x 10-1 

and 

b) 


a  = 


3  - 


(1  -  e 


- .  30461826164"ir37 

— ( 1  “  s  J 


=  8.346957421 


For  this  data  set 

l  =  8.34695742 
b  =  3.0461826104x10 


-1 


. 305t. 
1) 
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Performance  Measures 

After  estimation  of  the  parameters  a  and  b  the  next 
step  is  to  obtain  performance  measures  as  follows: 

•  Expected  Number  of  Faults  Detected  by  Time  t  E[N(t)] 


E [ N ( t ) ]  is  obtained  using  the  following  formula: 

A 

E  [N  (t )  ]  =  a (l-e-bt) 

Let  t=10  then  substituting  values  for  a,b,  and  t 
in  the  above  formula  we  get 
E  [N ( 10 ) ]  =  8.34695742x(l-e 
=  7.950397851. 

•  Expected  Number  of  Remaining  Faults  by  Time  1 1  E[H(t)i 


-3 . 04  61826104xl0_1xl0) 


E[N(t)]  is  obtained  using  the  following  formula 
E  [N  ( t )  ]  =  ae-t>t  . 

Let  t=10  then  substituting  values  for  a,b,  and  t 
in  the  above  formula  we  get 

E [N ( 10) ]  =  8-3469742x(e'3‘0461826104x10  lxl°) 

=  8.34695742xe'3*0461826104 
=  0.3965595689 

A 

•  Reliability  at  Time  t  after  nth  testing  Interval  Rn(t) 


Rn(t)  is  obtained  from  the  following  formula 
Rn(t)  =  exp [-a{exp [-bxn] -exp [-bx (n+t) ] } ] 

For  the  above  example  n=3.  Let  t  be  1 .  After  sub- 

A  A 

stituting  values  of  a,  b,  n,  and  t  in  the  above  formula 
we  get 

r3(1)  =  exp[-8.34695742x{exp[-0. 30461826104x3] 

-exp t-0. 30461826104x(3+l) ] }  =  0.4152494843 
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•  Mean  Time  to  Failure  After  nth  Testing  interval  MTTFn 
MTTFn  is  obtained  using  the  following  formula 

MTTF  =  — —  =  - - - 

n  X (n)  abexpl-bxn] 

For  the  above  example  n=3  and  substituting  values  for 

A.  A 

a  and  b  in  the  above  formula  we  get 

MTTF,  =  - - - 

8 • 34695 74 2x0. 304 61826104xexp [-0.30461826104x3] 

=  .9808216948 
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8.  CONCLUDING  REMARKS 

1.  The  objective  of  this  report  was  to  present  information 
relevant  to  the  selection  and  use  of  an  analytical  model 
for  software  reliability  assessment.  Towards  this  goal, 
we  presented  a  summary  of  the  available  models  in  Sections 
3,  4,  and  5  which  was  backed  up  by  detailed  information 

in  Appendices  C,  D,  and  E. 

2.  An  important  step  in  selecting  a  model  is  to  develop  a 
framework  for  describing  the  software  development  en¬ 
vironment  which  can  be  mapped  onto  a  set  of  assumptions 
that  are  consistent  with  those  of  the  selected  model  or 
models.  This  is  a  difficult  task  and  requires  a  clear 
understanding  of  the  development  environment  as  well  as 
of  the  model  assumptions.  We  feel  that  the  material  in 
Section  6  should  be  of  great  help  in  accomplishing  this. 

It  should,  however,  be  pointed  out  that  the  assumptions 
required  to  formulate  and  develop  an  analytical  model 
are  rarely,  if  ever,  satisfied  in  the  real  world.  In 
other  words,  perfect  adherence  to  model  assumptions  is 
an  ideal  which  cannot  be  met  in  practice.  A  realistic 
approach,  then,  is  to  select  a  reliability  model  which 
captures  the  essence  of  the  modelled  environment. 

3.  It  should  also  be  pointed  out  that  the  arguments  pre¬ 
sented  in  Section  6.1  about  the  assumptions  and  in 
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Section  6.2  about  the  applicability  of  the  models 
are  based  on  our  interpretation  of  the  phases  in  the 
software  development  process.  If  particular  situations 
differ  from  the  scenarios  presented  there,  appropriate 
modifications  must  be  made  in  the  model  selection  process. 
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ADpendix  A 

SOFTWARE  ERRORS:  THEIR  SOURCES  AND  CLASSIFICATION 


A . 1  Sources  of  Software  Errors 

Software  (also  called  program)  is  essentially  an 
instrument  for  transforming  a  discrete  set  of  inputs  into 
a  discrete  set  of  outputs  (see  Figure  A.l).  It  comprises 
of  a  set  of  coded  statements  whose  function  may  basically 
be  one  of  the  following: 

1.  Evaluate  an  expression  and  store  the  result 
in  a  temporary  or  permanent  location. 

2.  Decide  which  statement  to  execute  next. 

3.  Perform  innut/output  operations. 

Since,  to  a  large  extent  software  is  produced  by  humans,  the 
finished  software  product  is  often  imperfect.  It  is  imper¬ 
fect  in  the  sense  that  a  discrepancy  exists  between  what 
the  software  can  do  versus  what  the  user  or  the  omputing 
environment  wants  it  to  do.  The  computing  environment  refer 
to  the  physical  machine,  operating  system,  compiler  and 
translators,  utilities,  etc.  These  discrepancies  are  what 
we  call  software  errors  (see  Figure  A. 2) .  Basically,  the 
software  errors  can  be  attributed  to  the  following: 

1.  Ignorance  of  the  user  requirements, 

2.  Ignorance  of  the  rules  of  the  computing 
environment,  and 


Fig,  a. i .  Functional  View  of  Software 
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3.  Poor  communication  of  software  requirements 
between  the  user  and  the  programmer  or  poor 
documentation  of  the  software  by  the  programmer. 
The  fact  of  the  matter  is  even  if  we  know  that  soft¬ 
ware  contains  errors,  we  may  not  know  with  certainty  the 
exact  identity  of  these  errors. 

Currently,  there  are  two  major  paths  one  can  follow 
to  expose  software  errors: 

1.  Program  proving,  and 

2.  Program  testing. 

Program  proving  is  more  formal  and  mathematical  while 
program  testing  is  more  practical  and  still  remains  to 
be  heuristic  in  its  approach.  The  approach  in  program 
proving  is  the  construction  of  a  finite  sequence  of  logi¬ 
cal  statements  ending  in  the  statement  Usually  the  output 
specification  statement)  to  be  proved.  Each  of  the  logi¬ 
cal  statements  is  an  axiom  or  is  a  statement  derived  from 
earlier  statements  by  the  application  of  an  inference 
rule.  Program  proving  making  use  of  inference  rules  is 
known  as  the  Inductive  Assertion  Method.  This  method  was 
mainly  popularized  by  Floyd,  Hoare,  Dijkstra  and  recently 
Reynolds.  Other  work  on  program  proving  is  the  work  on 
the  Symbolic  Execution  Metnod.  This  method  is  the  basis 
of  some  automatic  program  verifiers.  Despite  the  formalism 


and  mathematical  exactness  of  program  proving,  it  is 
still  an  imperfect  tool  for  verifying  Drogram  correct¬ 
ness.  Gerhart  and  Yelowitz  [GER  76]  showed  several 
programs  which  were  proven  to  be  correct  but  still  con¬ 
tained  errors.  The  errors  were  due  to  failures 
in  defining  what  exactly  to  prove  and  were  not  failures 
of  the  mechanics  of  the  proof  itself. 

Program  testing  is  the  symbolic  or  physical  execu¬ 
tion  of  a  set  of  test  cases  with  the  intent  of  exposing 
embedded  errors  (if  any)  in  the  program.  Like  program 
proving,  program  testing  remains  an  imperfect  tool  for 
verifying  program  correctness.  A  given  testing  strategy 
is  good  for  exposing  certain  kinds  of  errors  but  not  all 
possible  kinds  of  errors  in  a  program.  An  advantage  of 
testing  is  that  it  provides  accurate  information  about  a 
program's  actual  behavior  in  its  actual  computing  environ¬ 
ment;  proving  is  limited  to  conclusions  about  the  program's 
behavior  in  a  postulated  environment. 

Neither  proving  nor  testing  can,  in  practice,  guaran¬ 
tee  complete  confidence  on  the  correctness  of  programs. 

Each  has  its  pluses  and  minuses.  They  should  not  be 
viewed  as  competing  tools.  They  are, in  fact,  complementary 
methods  for  decreasing  the  likelihood  of  program  failure 
(GOO  77] . 
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A. 1 .  SOFTWARE  ERROR  CLASSIFICATION 

A  systematic  study  of  software  errors  in  a  program 
requires  knowing  what  specifically  these  errors  are  and 
knowing  which  tool(s)  to  use  to  expose  particular  types 
of  software  errors.  Software  errors  can  be  grouped  as 
syntax,  semantic,  runtime,  specification  and  performance 
errors. 

A. 2.1  Syntax  Errors 

These  errors  are  due  to  discrepancies  between  the 
program  code  and  the  syntax  rules  governing  the  parser 
or  lexical  analyzer  of  a  program  translator.  These  are 
the  easiest  errors  to  detect.  They  can  be  detected  by 
visual  inspection  of  the  code  or  can  be  detected  mechani¬ 
cally  during  the  program  compilation  process.  Experienced 
programmers  rarely  commit  syntax  errors. 

A. 2.2  Semantic  Errors 

These  errors  are  due  to  discrepancies  between  the 
program  code  and  what  the  semantic  analyzer  of  the  computing 
environment  accepts.  Among  the  popular  kinds  of  semantic 
errors  are  typechecking  errors  and  implementation  restric¬ 
tion  errors.  Again,  they  may  be  detected  by  the  semantic 
analyzer  of  a  program  translator  or  by  visual  inspection. 
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Syntax  and  semantic  errors  are  detected  during  the 
compilation  stage  of  a  prooram.  A  program  having  syntax 
and/or  semantic  errors  cannot  be  executed.  Syntax  and 
semantic  errors  are  mainly  due  to  the  ignorance/neqliqence 
on  the  part  of  the  programmer  about  the  restrictions  and 
limitations  of  the  language  (s)he  is  using. 

A. 2. 3  Runtime  Errors 

As  the  name  implies,  runtime  errors  occur  during 
the  actural  running  of  a  program.  They  may  be  further 
classified  into  three  categories: 

Domain  errors 

A  domain  error  occurs  whenever  the  value  of  a 
program  variable  exceeds  its  declarea  range  or  exceeds 
the  physical  limits  of  the  hardware  representing  the 
variable.  The  declared  range  of  a  variable  is  done  im¬ 
plicitly  or  explicitly.  FORTRAN,  for  example,  assigns 
types  to  variables  based  on  the  variable  name  or  based 
on  a  declaration  statement.  PASCAL  requires  all  variables 
to  be  explicitly  declared  in  a  declaration  statement. 
PASCAL  has  facilities  to  declare  ranges  by  enumeration 
and/or  subsets  of  numeric  domains. 

Some  program  translators  produce  runtime  code  for 
checking  certain  types  of  domain  errors.  Some  have  built- 
in  recovery  features  for  domain  errors  (e.g.  PL/1,  COBOL) 
and  others  (e.g.  FORTRAN)  simply  abort  execution  upon 
the  occurrence  of  a  domain  error.  Certain  compilers,  like 
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PASCAL,  automatically  check  for  values  outside  a  declared 
range . 

Domain  errors  are  a  serious  matter  because 

a)  program  execution  is  aborted,  and/or 

b)  program  results  are  incorrect. 

Execution  abortion  may  be  fatal  especially  in  real-time 
systems.  Despite  their  seriousness,  domain  errors  have 
never  been  formally  and  extensively  studied  in  the  litera¬ 
ture.  This  is  because  detection  of  domain  errors  can  be 
very  difficult.  They  require  exact  specification  of  the 
ranges  of  the  input  variables.  Also,  the  test  values 
required  to  expose  these  errors  may  occur  at  the  input 
domain's  boundary  or  inside  the  input  domain  itself. 


Computational  errors 

Computational  errors,  sometimes  known  as  logic  errors, 
result  whenever  the  program  results  in  an  incorrect  output. 

The  incorrect  output  may  be  due  to  a  wrong  formula,  an 
incorrect  control  flow,  assignment  to  a  wrong  variable, 
incorrect  parameter  passing,  etc. 

It  is  not  possible  to  generate  runtime  code  to  de¬ 
tect  computational  errors  during  program  execution.  This 
is  because  computational  errors  are  really  discrepancies 
between  the  program's  output  and  the  program's  specifica¬ 
tions  . 

i 


Computational  errors  due  to  incorrect  program  con¬ 
structs  and  statements  may  be  detected  by  any  of  the 
structure  dependent  or  structure  independent  testing 
techniques.  However,  none  of  these  tools  can  guarantee 
total  absence  of  these  types  of  computational  errors  in 
a  program.  Computational  errors  due  to  missing  program 
constructs  and  statements  may  be  detected  by  any  of  the 
structure  independent  testing  techniques.  Again,  nore 
of  these  tools  can  guarantee  total  absence  of  comnutational 
errors  due  to  missing  paths. 

Non-Termination  errors 

Non-termination  error  is  simply  the  failure  of  a 
program  to  terminate  in  finite  time  without  outside  inter¬ 
vention.  The  most  common  cause  of  non- termination  errors 
is  when  the  program  runs  into  an  infinite  loop.  Mon¬ 
termination  can  also  occur  if  a  set  of  concurrent  programs 
falls  into  a  dead  lock. 

Infinite  loops  are  detected  by  simply  executing 
each  of  the  loops  in  a  program.  However,  this  strategy 
may  not  guarantee  total  absence  of  infinite  loops.  Some 
infinite  loops  may  only  occur  if  certain  program  variables 
achieve  certain  values.  Program  proving  may  also  be  used 
on  certain  programs  to  expose  infinite  loops.  The  problem 
of  program  non- terminat ion  in  general  is  still  an  unsolved 
problem. 


A. 2 . 4  Specification  Errors 


Specification  errors  result  whenever  there  exists  a 
discrepancy  between  the  statement  of  specifications  and 
the  statement  of  user  requirements.  A  requirements  error 
exists  whenever  there  is  a  discrepancy  between  the  state¬ 
ment  of  user  requirements  and  the  real  user  requirements. 

Presently,  detection  of  specification  errors  such  as: 

1.  Incomplete  specifications, 

2.  Inconsistent  specifications,  and 

3.  Ambiguous  specifications, 

remains  an  informal  process.  This  is  mainly  due  to  the 
nonexistence  of  a  specification  language  powerful  enough 
to  translate  the  user  requirements  into  clear,  complete 
and  consistent  terms. 

A  testing  tool  to  detect  specification  errors  is 
yet  to  be  developed. 

A-2.5  Performance  Errors 

Performance  errors  exist  whenever  a  discrepancy 
exists  between  the  actual  performance  (efficiency)  of  the 
programs  and  its  desired  or  specified  performance.  Program 
performance  may  be  measured  in  a  number  of  ways: 


1. 

Response 

time 

2. 

Elapsed 

time 

3. 

Memory  space  usage 

4. 

Working 

set  requirement,  etc. 
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The  actual  measurement  of  the  above  measures  of 
program  performance  can  be  a  very  difficult  process. 

Program  complexity  theory  tries  to  estimate  bounds  on 
the  running  time  of  certain  program  algorithms.  Statis¬ 
tical  analysis  and  simulation  can  also  be  employed  to 
estimate  the  above  performance  variables.  However,  use 
of  these  tools  can  be  very  expensive  and  time  consuming. 

A  performance  testing  tool  that  is  economical  (time- 
wise  and  costwise)  to  use  is  yet  to  be  developed. 

The  most  expensive  kind  of  software  errors  to  elimi¬ 
nate  are  those  which  are  not  discovered  until  late  in  the 
software  development,  such  as  when  the  software  becomes 
operational.  These  are  known  as  persistent  software  errors. 
Glass  [GLA81]  reported  that  persistent  software  errors  are 
mostly  due  to  the  failure  of  the  problem  solution  (i.e.  the 
proaram)  to  match  the  complexity  of  the  problem  to  be 
solved  (i.e.  the  user  requirements).  Examples  of  such 
errors  are  computational  errors  due  to  missing  or  insuffi¬ 
cient  predicates  and  failure  to  reset  a  variable  to  some 
baseline  value  after  its  use  in  a  functional  logic  segment. 
The  solution  to  this  software  problem  is  beyond  the  current 
state-of-the-art. 
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Appendix  B 

BASIC  RELIABILITY  CONCEPTS 

In  this  appendix  we  present  some  of  the  basic  concepts 
from  reliability  theory  that  are  useful  in  understanding 
the  software  reliability  models.  For  a  detailed  discus¬ 
sion  of  this  material,  the  reader  should  refer  to  a  standard 
book  such  as  Barlow  and  Proschan  [BAR75]  or  Mann  et  al  [MAN74]. 

In  the  following  we  will  use  the  random  variable  X 
to  denote  the  time  to  failure  and  x  to  denote  a  realiza¬ 
tion  of  X. 

Cumulative  Distribution  Function  (cdf) 

The  cdf  of  a  random  variable  X  is  denoted  by  F(x)  and 
represents  the  probability  of  X  being  less  than  or  equal  to 
x,  i . e  .  , 

F(x)  -  P(X  <  x) 

Probability  Density  Function  (pdf) 

The  pdf  of  X  is  denoted  by  f(x)  and  is  defined  as 
follows : 

f(x)  =  35c  F(x) 

Reliability 

The  term  reliability  refers  to  the  probability  of 
failure  free  operation.  Specifically,  reliability  R(x) 
is  defined  as  follows 
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R(x)  =  P(no  failure  by  x) 
or  R(x)  =  P(X  >  x) 

This  is  an  important  and  comm  ally  used  measure  of  a 
software  system. 


Hazard  Function  or  Failure  Rate 

This  is  a  very  fundamental  term  in  software  (or  hard¬ 
ware)  reliability  modelling  work.  The  hazard  function  is 
defined  as 


z(x) 


f(x) 

1  -  F (x) 


It  is  a  measure  of  the  instantaneous  speed  of  failure. 


Reliability  in  Terms  of  Hazard  Function 

Given  the  hazard  function,  the  software  reliability 
can  be  computed  from  the  following  relationship . 

-/*  z (u) du 
R(x)  =  e  U 


Mean  Time  to  Failure  (MTTF) 

This  measure  represents  the  expected  value  of  the 
time  to  next  failure  and  is  a  very  commonly  used  measure 
of  software  quality.  It  can  be  computed  by  any  of  the 
following  formulae 


MTTF 


j  xf(x)dx 
0 


MTTF 


,00 

R(x) dx 

0 
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Appendix  C 


nr.TATl.S  OF  TIMES  BETWEEN  FAILURES  (TBF)  MODELS 

This  appendix  contains  theoretical  and  computational 
details  of  most  of  the  TBF  models  discussed  in  Section  3. 
The  following  information  is  provided  for  each  model: 

.  Model  Assumptions 
.  Rasic  Formulae 
.  Parametric  Estimation 
.  Performance  Measures 
.  Data  Requirements 
Model  Applicability 
.  Relevant  References 


C-l  Jelinski  and  Moranda  De-eutrophication  Model  (Model  TBF1' 


1.  Assumptions 


a.  Initial  fault  content 

.  An  unknown  fixed  constant  N 

b.  Independence  of  faults 

.  Each  fault  in  the  program  is  independent  of  other 
faults  and  each  of  them  is  equally  likely  to  cause 
a  failure  during  testing.  Times  between  occurrences 
of  faults  are  independent  of  each  other. 

c.  Fault  removal  process 

.  A  detected  fault  is  removed  with  certainty  at  the 
end  of  each  testing  interval 

.  Only  one  fault  is  removed  during  each  testing  interval 
The  fault  removal  time  is  negligible 
.  No  new  faults  are  introduced  during  the  fault  removal 
process . 

d.  Hazard  function 

The  software  failure  rate  of  the  hazard  function 
during  a  failure  interval  is  constant  and  is  pro¬ 
portional  to  the  current  fault  content  of  the 
tested  program.  Thus,  during  the  ith  testing 
interval , 

zftj)  =  *{N  -  ( i  - 1)  } , 

where  q  is  a  oroportionalitv  constant,  and  N  is 
the  initial  number  of  faults  in  the  prop,ram. 
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2.  Basic  Formulae 


The  time  between  the  (i-l)st  and  the  ith  failures  T^  , 
is  distributed  exponentially  with  parameter  <|>[N-  (i-1)]. 
pdf  of  :  f(ti)  =  <J)  [N  -  (i-1)  ]exp[ -4»{N  -  (i-l]}t^] 

cdf  of  Ti:  F(t.)  =  1  -  exp  { -  $  {N  -  (i-ljt^] 

Plots  of  pdf  and  cdf  for  N  =  100  and  <J>  =  .01  are  shown  if 
Figure  D-l. 

3 .  Estimation  of  Parameters  and  Related  Results 

A  series  of  n  failures  is  observed  with  interfailure 
times  as  t t? , . . . , t  .  Usually,  the  method  of  maximum  like¬ 
lihood  is  used  to  estimate  the  parameters  N  and  <J>  as  shown 
below. 

The  likelihood  function  of  N  and  <j>  is 

n 

L(N,4»|t1,t2,  .  .  .  ,tn)  =  ^  d>  [N  -  (i-1)  ]  exp  t  -  d>  { N  -  (i-l)}^] 


The  maximum  likelihood  estimates  (mle's)  of  N  and  4>  are 
obtained  by  solving  the  following  pair  of  equations  simultaneous¬ 
ly. 


n 

E 

i=l 


1 

N-(i-l) 


n 

Z  <P  t. 
i=l 


0 


and 


—  -  E  [N-  (i-1) ] t •  =  0 


n 


i=l 

The  above  two  equations  yield 


n 

I 


i=lN-(i-1)  „  1 


n 


E  (i-l)t. 
1  i=l 


n 


where  T  =  E  t. 

i=l  3 


We  can  solve  the  above  equation  numerically  to  find  N  and 

A 

then  obtain  4>  from  the  following  equation 

n 


<f> 


NT  -  l  (i-l)t. 
i  =  1  1 


Using  the  asymptotic  properties  of  the  maximum  like 
lihood  estimators,  it  can  be  shown  that 

1 


Var(N)  = 


n 


det  A 


where 


Var  (4> )  -  *  det  A 

Cov  (N ,  $  )  -  -aii"A 

T<t> 

pn,$  "  "  /z .  ' 

det  A  =  \  . 1  (N^I+I) 2  '  t2, 

$  i=l 


and 


T2  =  E 


i=l  (N-i+1) 


4 .  Performance  Measures 

•  Reliability  at  time  t  after  the  nth  failure  is 

Rn(t)  =  P(Tn+1  >  t)  =  exp[-4>  {N-n}t] 

•  Mean  Time  to  Failure  (MTTF)  after  n  failures  is 

MTTFn  “  £Vt,dt  '  ?  -[i-nT 

Plots  of  RjQ(t)  and  of  MTTF  after  1,2,  ,*#,25  failures  are 
shown  in  Figure  02. 
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A  A 

Using  the  variance-covariance  of  N,4i  ,  it  can  be  shown  that 

—  2 4> i  (N— n)  t  t  o  2  2i 

Variance CRn(t))  =  6  -  -  {nt2-2  (N-n) Tt  <0  +  (N-n)  t  I2> 

Variance  (MTTFn)  =  “  2Tviv2  +  ^2l2}  ' 


where 


V1  =  ‘ 


(N-n)  <f> 


v_  - - j  ,  and 

(N-n)* 

Z2  and  det  A  are  as  defined  before. 

All  the  above  quantities  can  be  computed  by  replacing 

/A  -A 

N  and  4>  by  N  and  *  ,  respectively. 

5 .  Data  Requirements 

Data  on  times  between  failures,  ^'**2  '  ‘  * 'fcn  are  required 
to  estimate  the  parameters  N  and  *  (see  #  3  above) . 

6 .  Model  Applicability 

If  the  underlying  assumptions  are  satisfied,  this  model 
can  be  employed  during  the  system  integration  testing  phase. 

7 .  Relevant  References 

l«JEL72]'Was  the  original  paper  in  which  this  model  was  in¬ 
troduced.  Further  discussion  and  some  additional  details  are  in 
[MOR75,  MORE  1].  Computations  of  various  relevant  quantities  and 
appropriate  confidence  bounds  are  discussed  in  [GOE79,a,GOE80  ] . 
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02.  Schick-Wolverton  Linear  Model  (Model  TBF2) 


1 .  Assumptions 

a.  Initial  fault  content 

.  An  unknown  fixed  constant  N 

b.  Independence  of  faults 

.  Each  fault  in  the  program  is  independent  of  other 
faults  and  each  of  them  is  equally  likely  to  cause 
a  failure  during  testing.  Times  between  occurrences 
of  faults  are  independent  of  each  other. 

c.  Fault  removal  process 

.  A  detected  fault  is  removed  with  certainty  at 
the  end  of  the  debugging  interval 

Only  one  fault  is  removed  during  each  testing  interval 

.  The  fault  removal  time  is  negligible 

Mo  new  faults  are  introduced  during  the  fault 
removal  process 

d.  Hazard  function 

.  The  software  failure  rate  or  the  hazard  function, 
at  any  time  is  proportional  to  the  current  fault 
content  of  the  program  and  to  the  time  elapsed 
since  the  last  failure.  Thus,  the  hazard  function 
between  the  (i-l)st  and  the  ith  failures  is  given 
by 

z(t.)  =  <f> [ N  -  (i-l)]ti 

where  0  is  a  proportionality  constant,  N  is  the 
initial  fault  content,  and  ti  is  the  test  time  since 


l 


2.  Basic  Formulae 


The  time  between  the  (i-l)st  and  the  ith  failures 

has  a  P.ayleigh  distribution  with  scale  parameter  equal  to 
1/2 

{${N  -  (i-l)}/2>  .  Note  that  Rayleigh  is  a  special  of 

the  Weibull  distribution  with  shape  parameter  equal  to  2. 
pdf  of  T.:  f(t.)  =  4>  [N  -  (i-1)  ]t.expt-<HN  -  (i-l)]t?/2] 

ti 

cdf  of  T.  :  F(t-)  =  1-  exp[  -  4>  C  N  -  (i-1)]  /  t  dt] 

0 

=  1  -  exp  [  - <t>  (N  -  Ci-  1)  1 1?/2] 

The  pdf  of  each  failure  interval  is  a  different  Rayleigh 
distribution  and  each  is  an  Increasing  Failure  Rate  (IFR) 
distribution  [BAR7  5  , C1OE8O  ]  .  Plots  of  pdf  and  cdf  for 
N  =  150,  <p  -  .02  are  shown  in  Figure  C-3. 

3 .  Estimation  of  Parameters 

A  series  of  n  failures  is  observed  with  interfailure 
times  as  t^» tj, . . . , t  ,  Usually  the  method  of  maximum  like¬ 
lihood  is  used  to  estimate  the  parameters  N  and  <t> ^  as  shown 
below. 

The  likelihood  function  of  N  and  <J>  is 


L(N,$  |t,,t  ,...t)  =  n  4>  [N-  (i-1)  ]  t .  exp  [  -<f>  [N-(i-l)  ]t2/2] 
c  11  i=l  1  1 

The  maximum  likelihood  estimates  of  parameters  N  and  <j> 
are  obtained  by  solving  the  following  pair  of  equations  simul¬ 
taneously. 

—  -  E  (N-(i-l)  )t?  =  0 
<t>  i=l  1 


The  above  two  equations  yield 


n 

E 

i=l 


1 

N-(i-l) 


n  j  n  - 

N  E  tf  -  E  (i-l)t: 
i=l  1  i=l  1 


We  can  solve  the  above  equation  numerically  to  find 
and  then  obtain  4»  from  the  following  equation 


Plots  of  R5Q(t)  and  MTTF  after  1,2, 3, 4,  and  5  failures 
are  shown  in  Figure  C-4. 
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RELIABILI 


MTTF  AFTER  1,2 . 5  FAILURES 

Figure  C.4  Plots  of  Reliability  and  MTTF  for  Model  TBF2 


5.  Data  Requirements 


Data  on  times  between  failures  t^/tj/ • . • /tn  are  required 
to  estimate  the  parameters  N  and  $2  (see  #4  above) . 

6,  Model  Applicability 

Here  times  between  failures  are  assumed  to  follow  IFR  dis¬ 
tributions  and  hence  this  model  should  be  used  only  when  the 
testing  strategy  justifies  such  increasing  fault  detection  rate. 

7 .  Relevant  References 

The  model  was  first  proposed  by  Schick  and  Wolverton  [SCH72] , 
A  comparative  study  with  other  models  was  reported  in  [SCH78]; 
however,  some  of  the  material  in  that  paper  could  be  misinter¬ 
preted  as  pointed  out  in  [GOE80] . 
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.  Geometric  De-eutrophication  Model  (Model  TBF4) 

Assumptions 

a.  Initial  fault  content: 

.  No  specific  value  needs  to  be  assumed 

b.  Independence  of  fault: 

.  Fatal  faults  are  independent  of  each  other  and  each 
of  them  is  equally  likely  to  cause  a  failure  during 
testing.  Times  between  failures  are  independent  of 
each  other. 

c.  Fault  removal  process: 

.  Testing  is  carried  out  until  a  fatal  fault  occurs 
and  then  the  accumulated  group  of  faults  is  re¬ 
moved  along  with  the  fatal  fault. 

.  A  fatal  fault  (possibly  with  other  non-fatal 
faults)  is  removed  with  certainty  at  the  end 
of  each  testing  interval 

.  The  fault  removal  time  is  negligible 

.  No  new  faults  are  introduced  during  the  fault 
removal  process 

d.  Hazard  function: 

.  The  software  failure  rate  or  the  hazard  function 
during  a  testing  interval  is  a  constant  but 
changes  values  at  occurrences  of  failures.  For 
the  ith  such  interval,  the  hazard  function  is 
given  by 

z(t.)  =  Dk1'1 

where  D  is  the  fault  detection  rate  during  the 
first  interval  and  k  is  a  constant,  0  <  k  <  1. 
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2 .  Basic  Formulae 

Time  between  the  (i-l)st  and  ith  failures,  T^,  is  dis¬ 
tributed  exponentially  with  parameter  Dk1  ^ 

pdf  of  T^:  f(t^)  =  Dk^"  *  exp[-Dk* 

cdf  of  Ti:  F ( t i }  =  1  -  exp{-Dk1'1ti ] 

Plots  of  pdf  and  cdf  for  D  *  0.5,  k  *  0.95  are  shown  in  Figure  D- 3 

3 .  Estimation  of  Parameters  and  Related  Results 

A  series  of  n  failures  is  observed  with  interfailure 
times  as  •  Usually  the  method  of  maximum  likelihood 

is  used  to  estimate  the  parameter  D  and  k  as  shown  below. 

The  likelihood  function  of  D,k  is 

L(D,k|  t,  ,t9,  .  .  .  ,t  )  =  ir  Dk1  expl-Dk1  ^t .  ] 

i=l 

The  maximum  likelihood  estimators  of  parameters  D  and  K  can 
be  obtained  by  solving  the  following  pair  of  equations  simul¬ 
taneously. 


2-  -  E  ki_1  t.  =  0 
d  i-1  1 


1  n  n  i-2 

~  I  (i-l)-D  Z  (i-1 ) k  *t.  =  0 

K  i=l  i=l 


The  above  two  equations  yield 


Iiklti  _  n+1 
Zk1ti  2 


from  which  we  can  find  k  by  solving  the  equation  numerically, 

A 

and  then  D  from  the  following  equation 


"  -i-1 

Z  k  t 
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Using  the  asymptotic  properties  of  the  maximum  likelihood 


estimators,  it  can  be  shown  that 


v~<s.  -  °2 


Var(k)  =  k 


12 


Cov(.D,k)  =  -Dk 


n (n  -1) 
6 


n  (n+1 ) 


Pf»  C  =  ~|/3  (n-l) 
U,K  2  (2n-l) 


4.  Performance  Measures 

•  Reliability  at  time  t  after  the  nth  failure  is 

Rn(t)  3  P(Tn+1  >  t)  =  exp[-Dknt] 

•  Mean  Time  to  Failure  (MTTF)  after  n  failures  is 

OO 

MTTF  =  /R_  (t)  *  ~- 
n  _  n  rM,n 


Plots  of  R2j(t)  and  MTTF  after  1,2,  111,20  failures 
shown  in  Figure  D-3. 

Using  the  variance-covariance  of  D,  £  it  can  ce 

that 


Var(MTTFn) 


3n  +  7) 
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are 


shown 


5.  Data  Requirements 

Data  on  times  between  fatal  failures  ti't2,’**'tn  are 
required  to  estimate  the  parameters  D  and  k  (see  #4  above) . 

6.  Model  Applicability 

This  model  can  be  used  when  testing  continues  until  the 
occurrence  of  a  fatal  fault  at  which  time  the  fatal  and  the 
non-fatal  faults  are  removed. 

7.  Relevant  references 

The  model  was  proposed  by  Moranda  in  [MOR75] .  This  model 
was  used  to  analyze  some  failure  data  and  to  compare  the 
results  with  some  other  models  in  [SUK76] . 
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Appendix  D 

DETAILS  OF  FAILURE  COUNT  MODELS 


In  this  appendix  we  present  detailed  technical  material 
about  the  failure  count  models  of  Section  4.  The  following 
models  are  discussed: 

D-l  Goel-Okumoto  Non-Homogeneous  Poisson  Process  Model 
(Model  FC1 ) 

D-2  Goel  Modified  Non-Homogeneous  Poisson  Process  Model 
(Model  FC3) 

D-3  Musa  Execution  Time  Model 
(Model  FC4 ) 

D-4  Shooman  'lode  1  (Model  FC5) 

D-5  Geometric  Poisson  Model  (Model  FC6) 

D-6  Modified  Je 1 inski -Moranda  Model  FC7) 

D-7  Modified  Geometric  De-Eutronhication  Model 
(Model  FC8 ) 

D-8  Modified  Schick -Wolverton  Model  (Model  FC9) 

D-9  Generalized  Poisson  Model  (Model  FC10) 

D- 10  IBM  Binomial  Model  (Model  FC11) 

D-ll  IBM  Poisson  Model  (Model  FC12) 
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D- 1  Goel-Okumoto  Non -Homogeneous  Poisson  Process  Model 
(Model  FC1 ) 

1 .  Assumptions 

a.  Initial  fault  content: 

.  Expected  number  of  software  faults  to  be  eventually 
detected  is  an  unknown  fixed  quantity 
Actual  number  of  faults  to  be  observed  is  a 
random  variable 

b.  Independence  of  faults: 

Each  failure  is  caused  by  one  fault  and  each  of 
them  is  equally  likely  to  cause  a  failure  during 
testing. 

Number  of  software  faults  detected  during  non¬ 
overlapping  testing  intervals  is  independent  of 
each  other 

c.  Fault  removal  process: 

.  Fault  removal  time  is  negligible 

No  new  faults  are  introduced  during  the  fault 
removal  process. 

d.  Intensity  function: 

.  Expected  number  of  software  faults  detected 

during  (t,t  +  Atj  is  proportional  to  the  expected 
number  of  software  faults  undetected  by  time  t, 
i .  e. 

m(t+At)  -  m(t)  =  b(a  -  m(t)}At 

where 
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m(t)  =  expected  number  of  software  faults  detected 


by  time  t, 

a  =  expected  number  of  software  faults  to  be 
eventually  detected, 
b  =  constant  of  proportionality. 

The  intensity  function  or  the  fault-detection  rate 
A(t)  is  a  decreasing  function  of  t  and  is  given  by 
\(t)  =  m'  (t)  =  b{a-m(t)}  =  abe  bt 
where  a,b,  and  m(t)  are  as  defined  above. 

2 .  Basic  Formulae 

Expected  number  of  software  faults  detected  by  time  t 
is  given  by 

m(t)  =  a ( 1  -  e’bt) 

where  a  and  b  are  as  defined  above. 

The  total  number  of  software  faults  detected  by  time  t 
N(t),  under  the  above  assumptions  is  an  NHPP  with  mean  value 
function  m(t)  and  intensity  function  A(t)  as  given  above. 

The  distribution  of  N(t),  hence,  is  given  by 

P(N(t)  =  y}  e‘m(t),  y  =  0,1,2,... 

and 

E[N(t)  ]  =  m(t)  =  a (1  -  e'bt) 

3 .  Estimation  of  Parameters 

Suppose  y1,y2 ,  • . •  ,  yn  are  the  cumulative  number  of 
software  faults  detected  by  times  t .,  t t  ,  respectively 
Then  the  likelihood  function  for  (a,b)  given  the  data  pairs 


t (y i • t i ) »  i  =  1,2, . . . ,n}  is 

L(a,b  fy1,y2, . . . ,yn,t1,t2, . . .  ,tn)  = 

=  Pr{N(t2)  =  yi,N(t2)  *  y2,...,N(tn)  =  yn} 


n  [m(t. )  -  m(t.  ) ] 

=  n  - - - — 

1=1 


yi-ri-l 


- (m(ti) -m(ti_  x) } 


„  -bt..  -bt.  y-i-y-T  -  bt 

n  La(e_ M)  1  1'1  e-«d-e  ") 

i*l  (Vi  -  Xj-iJ ! 


MLE  of  parameters  a  and  b  can  be  obtained  by  solving  the 


following  pair  of  equations  simultaneously 
-bt 

a ( 1  *  e  n)  -  yn 


at  e'bt”  =  l  (yi  'yi-l)(tie  bt‘  ~  Vl6  bti']) 
n  i*l  -bt.^  -bt- 

e  -  e 


The  above  two  equations  yield 

-bt  -bt. 

yte  n  (y--y-i)(t.e  1  -  t .  ,( 

'  n  n  _  r  w  i  ’  i  - 1 1  K  i _  i-I 

-bt  ”  ^bt  i  '  -bt  - 

o  n,  11  „  l-l  „  l 

(i-ej  e  -e 


■bti-l. 


from  which  we  can  find  b  numerically  and  hence 


/  ■%  n  \ 

(1  -  e  ) 


4.  Performance  Measures 


.  Expected  number  of  software  faults  detected  by 
time  t  is  given  by 

A 

m(t]  =  a ( 1  -  e"bt) 


Expected  number  of  remaining  faults  in  the  s/w 


system  at  time  t  is  given  by 
E[N (t)]  = 

where  N(t)  =  number  of  faults  remaining  in  the  system 
at  time  t. 

.  Software  reliability 

Let  be  the  time  between  failures  (k-1)  and  k 
and  be  the  time  to  k  failures.  Then  it  can  be  shown  that 
the  conditional  reliability  function  of  X^ ,  given  ^  =  s, 
is 

Ry  re  (x  |s)  =  exp[-a{e*gs  -  e'6(s+x) }] 

*k,bk-l 

5 .  Data  Requirements 

The  data  needed  are  lengths  of  testing  intervals  and 
number  of  failures  in  each  interval.  Before  fitting  the  model 
however,  all  data  should  be  normalized  to  equal  test  intervals 

6 .  Model  Applicability 

Model  can  be  used  in  a  fairly  general  testing  environ¬ 
ment  (see  explanation  of  assumptions  in  Section  6.1). 

7 .  Relevant  References 

The  model  was  proposed  in  [GOE79a].  Detailed  data 
analyses  based  on  this  model  are  given  in  [GOE82]. 
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Goel  Modified  Non-Homogeneous  Poisson  Process 
Model  (Model  FC3) 

Assumptions 

a.  Initial  fault  content: 

.  Expected  number  of  software  faults  to  be  detected 
eventually  is  an  unknown  fixed  quantity. 

Actual  number  of  faults  to  be  detected  is  a 
random  variable. 

b.  Independence  of  faults: 

.  Each  failure  is  caused  b^  one  fault  and  each  of 
them  is  equally  likely  to  occur 
.  Number  of  software  faults  detected  during  non¬ 
overlapping  testing  intervals  is  independent 
of  others 

c.  Fault  removal  process: 

Fault  removal  time  is  negligible 
,  No  new  faults  are  introduced  during  the  fault 
removal  process 

d.  Intensity  function: 

Expected  number  of  software  faults  detected  during 
(t,t+At)  is  given  by 

m(t+At)  -  m(t)  =  bctCrl{a  -  m(t) }At 

where  m(t)  =  expected  number  of  software  faults 
detected  by  time  t, 

a  =  expected  number  of  software  faults 
to  be  eventually  detected 
b,c  =  constants. 
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The  intensity  function  or  the  fault  detection 


rate  A ( t )  is  a  function  of  time  and  is  given  by 

Q 

Aft)  =  m' (t)  =  abctc" 1e*bt 
where  a,b,c,  and  m(t)  are  as  defined  above. 

2 .  Basic  Formulae 

Expected  number  of  software  faults  detected  by  time  t 
is  given  by 

m(t)  =  a(l  -  e  bt  ^ 

where  a,b,  and  c  are  as  defined  above. 

The  total  number  of  software  faults  detected  by  time 
t,  Nft),  under  the  above  assumptions  is  a  NHPP  with  mean  value 
function  m(t)  and  intensity  function  Aft)  as  given  above. 

The  distribution  of  N(t)  ,  hence,  is  given  by 

PlN(t)-y)  •  Mtl£  y  .  0,1,2,... 

and  r 

E[N(t) 1  =  m(t)  =  a(l  -  e'bt  ) 

3 .  Estimation  of  Parameters 

Suppose  y2 ,y2, • • • ,yn  are  the  cumulative  number  of  soft¬ 
ware  faults  detected  by  times  t^  ,  t 2 , . . . , t  ,  respectively. 

Then  likelihood  function  for  (a,b,c),  given  the  data  pairs 
(y** tj)  ,  i  =  1,2,  .  .  .  ,n  is 

L(a,b,c  |y1,y2, . . . ,yn,t1,t2, . . . ,tn) 

Pr(N(t1)  =  y1,N(t2)  =  y2,...fN(tn)  =  yn) 
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n 

*  n 

i=  1 


fm(ti)  -  m(ti_ p ] 

‘  yk-l3  ! 


yi'yi-l 


-  (m(ti)-in(ti_1) } 


n 

=  n 
i  =  l 


-bt?  , 

{a(e  1-1  -  e 


-  bt? 


-bt 


21 _  e'a(1'e  } 


MLE  of  parameters  a,b,  and  c  can  be  obtained  by  solving 
the  following  non-linear  simultaneous  equations: 

-btn 

yn-aU-e 


fc  -K 

ate 

n 


-bt?  -bt? 

?  (yj  ~yi-i)(Iie  -ti-ie 

i*i  -bt?  -bt= 

(e  1  1  -  e  x) 


)  , 


and 


_  -bt; 

at^Un  tn)e 


-bt? 


I  Cyi~yi-lHtiC£nti)e  1-ti-l(£nt  i- 2 e 


i= 1  -bt?  •.  -bt? 

(e  1*1  -  e  1) 


The  set  of  simultaneous  equations  can  be  solved  numerically 
for  a,b,  and  c.  The  solution  will  be  the  required  maximum 

a  a  A 

likelihood  estimates  a,  b,  and  c  of  a,  b,  and  c,  respectively 
4 .  Performance  Measures 

.  Expected  number  of  software  faults  detected  by  time  t 
is  given  by 

-fitc 

m(t)  =  a(l  -  e  DX  ) 

.  Expected  number  of  remaining  faults  in  the  software 
system  at  time  t  is  given  by 

A 

~  hf C 

E [N (t)  ]  =  ae’ Dt 
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where  N (t)  =  number  of  faults  remaining  in  the  s/w 
system  at  time  t. 

.  Software  reliability 

Let  be  the  time  between  failures  (k-1)  and  k  and 
be  the  time  to  k  failures.  Then  the  conditional  relia¬ 
bility  function  of  X^,  given  by  1  =  s,  is 

Ry  (x|s)  -  exp(-5{e'bsC  -  e-bfs.xlc} , 

Ak|Sk-l 
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Musa  Execution  Time  Model  (Model  FC4) 


1.  Assumptions 

a.  Initial  Fault  Content: 

.  The  number  of  faults  in  the  program  (existing 
before  the  test  phase)  is  an  unknown  fixed 
quantity  N. 

b.  Independence  of  the  Faults: 

.  Faults  in  the  program  are  independent  of  each 

other  and  each  fault  has  a  constant  average  occur¬ 
rence  rate.  Failure  intervals  are  independent 
of  each  other. 

c.  Testing  Environment: 

Test  space  for  the  program  covers  the  use  space. 
The  set  of  inputs  for  each  run  of  the  program, 
whether  during  a  test  or  operational  phase,  is 
selected  randomly. 

d.  Fault  Removal  Process; 

.  New  faults  can  be  introduced  during  the  correc¬ 
tion  process. 

.  More  than  one  fault  can  be  corrected  during  each 
correction  time. 

Time  to  correct  faults  is  negligible. 

e.  Hazard  Function: 

.  Execution  times  between  failures  are  piecewise 
exponentially  distributed. 
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Hazard  rate  is  proportional  to  the  expected 
value  of  the  number  of  faults  remaining,  i.e., 

Z(t)  =  Kf (N  -  nQ) 
where,  Z  =  hazard  function 

t  =  execution  time  utilized  in  executing 
program  up  to  the  present 
N  *  initial  fault  content  (existing  before 
the  test  phase) 

nc  =  number  of  faults  corrected 
f  =  linear  execution  frequency  (average 
instruction  execution  rate  divided 
by  number  of  instructions  in  the 
program) 

K  *  proportionality  constant,  which  is 

an  fault  exposure  ratio  which  relates 
fault  exposure  frequency  to  linear 
execution  frequency. 

Basic  Formulae 

dnc 

Fault  correction  rate:  =  BCZ(t) 

where,  B  =  average  ratio  of  the  rate  of  reduction  of 
faults  to  the  rate  of  failure  occurrence 
(fault  reduction  factor)  . 

C  =  average  ratio  of  rate  of  detection  of 
failures  during  test  to  that  during  use 
(testing  compression  factor) . 


n 

2.  m  =  *  expected  number  of  failures  experienced  in  correctinq 

n  faults 

3.  M  =  g-  =  expected  total  number  of  failures  required  to 

expose  and  remove  the  N  inherent  faults 

4.  3?  +  BCfKm  =  BCfKM 


5.  m  =  M[1  -  exp  [-C'r/MTQ]  } 

where  TQ  =  initial  MTTF  before  testing. 

6.  n^  =  N [ 1  -  exp  [-ct/MTq ] ] 


3 .  Estimation  of  Parameters: 
Likelihood  Function 


“Vt2 . V  ’  ^  f0  U-  T~0  >1-  TT>! 


MLE  of  parameters  Tq  and  M  can  be  obtained  from  the  following 
pair  of  equations 

r  -  1  U-  =  0 

1Q  i=l  n  x 

.  0 
and 


m 


i=l  M- 


c 

MT 


m 

l  t.  =  0 
.  U  1 

0  1  =  1 


1.  Performance  Measures: 


Reliability:  R(t)  =  exo(- 


where 


T  =  MTTF  =  Tq  exp (ct/MTq) 
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D-4.  Shooman  Exponential  Model  (Model  FC5) 


1 .  Assumptions 

a.  Initial  Fault  Content: 

An  unknown  fixed  constant  N 

b.  Independence  of  faults 

Each  error  in  the  program  is  independent  of  other 
errors  and  each  of  them  is  equally  likely  to  occur. 

c.  Fault  Removal  Process: 

A  detected  error  is  corrected  with  certainty. 

No  new  errors  are  introduced  during  debugging 
Correction  time  of  an  error  is  negligible. 

d.  Hazard  Function 

Error  occurrence  rate  or  correction  rate  is 
proportional  to  the  number  of  remaining  errors 
nr(t),  i.e. 


where. 


:(t) 


i 


T 


n  ,  (  r  ) 


K 

t 


Knr(x) 

K{y  -  nc(T)  ] 

total  number  of  instructions  in 
the  program 

debugaing  time  since  start  of 
system  integration 
total  number  of  faults  corrected 
during  debugging  interval  t, 
normalized  with  respect  to  I. 
pronortionality  constant 
operating  time  of  rhe  system 
measured  from  its  initial  activation. 
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Basic  Formulae: 


Probability  that  a  s/w  failure  will  occur  in  time  interval 
(t,  t+At)  after  t  hours  of  successful  operation  is  proportional 
to  the  failure  rate  (hazard  function)  Z(t),  i.e. 

Pr (t  <  tf  <  t  +  At|tf  >  t}  =  Z (t) At  =  Knr(x)At 
where  t^  =  operating  time  to  failure 
Estimation  of  Parameters: 

Parameters  N  and  K  can  be  estimated  by  runnina  a  functional 

test  after  two  different  debugging  times,  \  ^  chosen  so 

that  n  (t.)  <  n  (r_).  Then 
cl  c  2 

MTTF1  -  r-  -  1  -  1*1?  -  w1'1 

si  1 

and 

MTTF2  -  rz  -  1^1  -  IKt?  - 

S  2  2 

where,  >  =  software  failure  rate 

H  =  total  number  of  run  hours  (successful) . 


4.  Performance  Measures: 


Reliability:  R(t)  =  exp[- 

=  exp[- 

MTTF :  MTTF  = 

\ 
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D- 5 .  Geometric  Poisson  Model  (Model  FC6) 

1 .  Assumptions 

a.  Initial  Fault  Content: 

No  specific  value  needs  to  be  assumed 

b.  Independence  of  the  Faults 

Each  fault  in  the  s/w  is  independent  of  others 
and  each  of  them  is  equally  likely  to  occur. 

c .  Error  Removal  Process: 

.  Number  of  faults  detected  in  any  debugging  interval 
(fixed)  are  removed  with  certainty  at  the  end  of 
the  debugging  interval. 

.  No  new  fault  is  introduced  during  the  correc¬ 
tion  time. 

Time  to  correct  the  detected  faults  is  negligible. 

d.  Hazard  Function: 

.  Debugging  intervals  are  fixed. 

.  The  number  of  faults  occurring  in  the  i-th  interval 
is  governed  by  a  Poisson  distribution  with 
parameter  XK1  i.e. 

Z(ti)  =  AKi_1, 

where , 

X  =  average  number  of  faults  occurring 
in  the  first  interval 
K  =  a  positive  number  less  than  1. 
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Basic  Formulae: 

The  distribution  of  the  number  of  faults,,  m.  ,  detected 


during  ith  debugging  interval 


is  Poisson  with  oarameter  \K 


i .  e .  , 


,  n . 
,„i-l.  1 


’ r ( N .  =  n  \  -  ('K*  A)  *  exp[-\Ki-1l 
1  i  n~;  - 

i 


E [Ni ]  =  \k 


i-1 


Estimation  of  Parameters 


Likelihood  Function: 


L(n, ,n,, . . . ,n  )  =  P r \N ,  =  n ,  ,  N  -  =  n  N 

m  1  l  £  l  m 


m 


-  ( ~« K 1  ~  ^ )  1  e  xp  [  -  a  K 1  ~  1  ] 


i=  1 


n .  : 
l 


Maximum  likelihood  estimators  of  parameters  '•  and  K  can  be 
obtained  by  solving  the  following  pair  of  equations  simultaneous 


1  ? 


m-1 


n .  - 


K1  =  0 


i=l 


i=0 


m-1  m-1 

\  '  iK1  =  >  in.,, 

i=0  i=0  1+1 


These  two  equations  yield 


m 


(1  Km)  (1  -  K) 


i=  1 


K  +  (m-l)Km+1  -  mKm 


m- 1 


in . 


i=0 


i+1 


from  which  we  can  find  K  and  hence 

m- 1 


/  in . 


i*0 


i+1 


m-1 


.  .'.i 


1A 


i  =  0 


d-  i: 


After  the  ith  debugging  interval  the  failure  rate  is 


XK1,  therefore 

i 

Reliability:  R(t)  =  Pr{no  failures  in  (  [  tm, 

nt=  1 


=  e 


-XK1t 


MTTF  after  i-th  debugging  interval 


j  R(t)  dt 
0 
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1.  Assumptions 


a.  Initial  fault  content: 

.  An  unknown  fixed  quantity  N 

b.  Independence  of  faults: 

Each  fault  in  the  program  is  independent  of  other 
faults  and  each  of  them  is  equally  likely  to 
cause  a  failure  during  testing. 

.  Number  of  faults  discovered  in  any  testing  interval 
is  independent  of  that  in  any  other  intervals. 

c.  Fault  removal  process: 

.  All  detected  faults  are  removed  with  certainty 
at  the  end  of  each  testing  interval 
.  Fault  removal  time  is  negligible 

No  new  faults  are  introduced  during  the  fault 
removal  process. 

d.  Hazard  function 

.  The  fault  occurrence  rate  or  the  hazard  function 


during  a  testing  interval  is  proportional  to  the 
number  of  remaining  faults  at  the  beginning  of 
this  interval  and  for  the  ith  testing  interval 
is  given  by 


where 


Z(t.)  =  <J> [N  -  M.^] 

N.  =  total  number  of  faults  removed 
1 

up  to  the  end  of  the  j th 
testing  interval. 


M. 

J 
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=  ith  testing  interval 
0  =  proportionality  constant 


2 .  Basic  Formulae 

Number  of  faults,  ,  detected  during  the  ith  testing 
interval  t^  has  a  Poisson  distribution  with  rate  <j>[N  -  NL  1] 
i .  e . 


Pr(N.  =  n. }  = 
1  i 


n.  .)t. 

{ 4>  (N  -  Mi_1)  ti}  e 


n-  i 


and 


E  [N^]  -  *(N  -  Mi_1) tL 


3. 


Estimation  of  Parameters 
Likelihood  Function 

L(n, ,n_, . . . ,n  )  =  Pr {N.=  n. ,N-  =  n_, . . . ,N  =  n  } 

12  m  112  2  mm 

ni 

m  U(N-M.  ,)t.} 

*  n  - — - -  exp  1-0  (N  -  M  )t.J 

i_l  n  i  J-  i 


MLE  of  parameters  N,0  can  be  obtained  by  solving  the  following 
pair  of  equations  simultaneously 


and 


m  n .  m 

Y  _ L_ 

. £,  N  -  M.  .  .  , 

1=1  l-l  1=1 


-  ♦  l  t,  =  0 


1  m  m 

?  Jx  ni  -  Jx  -  0 


The  above  pair  of  equations  yield 


n-20 


m  m 

(  l  n.)  (  I  t.) 
m  ni  i=l  1  i=l 

. '  ,  N  -  M.  ,  m 

1=1  i-l  1  (N-Mi_1)ti 

i=l 


from  which  we  can  find  N  and  therefore 

IU 

1  n* 
i=l  1 


m 

l 

i=l 


Performance  Measures: 

After  the  i-th  debugging  interval  the  failure  rate 
is  <f>  [N  -  Mi  ] 

Therefore 

i  i 

Reliability:  R(t)  =  Pr{no  failure  in  (  l  t  ,  [  t  +  t]  ) 

m-1  m  m-1  m 

-4>  (N-M±)  t 

=  e 

MTTF  after  i-th  debugging  interval 

00 

*  1  R(t,dt  - 
0  1 
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D- 7  Modified  Geometric  De - Eutrophication  Model  CModel  FC8' 


Assumptions 

a.  Initial  Fault  Content: 

.  No  specific  value  needs  to  be  assumed 

b.  Independence  of  faults: 

Each  fault  in  the  program  is  independent  of  other 
faults  and  each  of  them  is  equally  likely  to 
cause  a  failure  during  testing. 

.  Number  of  faults  discovered  in  any  testing  interval 
is  independent  of  that  in  any  other  intervals. 

c.  Fault  removal  process: 

.  All  detected  faults  are  removed  with  certainty  at 
the  end  of  each  testing  interval. 

Fault  removal  time  is  negligible 

.  No  new  faults  are  introduced  during  the  fault 
removal  process. 

d.  Hazard  function 

.  The  fault  occurrence  rate  or  the  hazard  function 
during  a  testing  interval  is  constant  but  the 
value  changes  at  the  beginning  of  the  next  testing 
interval.  For  the  ith  testing  interval,  t^,  the 
hazard  function  is  given  by 


Z(tt)  =  DK 


Mi-1 


where,  D  =  fault  detection  rate  during  the  first 

testing  interval  t1 
K  =  a  positive  constant  less  than  1 
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cumulative  number  of  faults  detected 


Mi-1 


up  to  the  end  of  (i-l)st  testing  interval. 


2.  Basic  Formulae 


Number  of  faults  N.  detected  in  the  ith  testing  interval 

1  Mi-1 

has  a  Poisson  distribution  with  rate  DK  ,  i.e. 


Pr{N.  =  n. }  = 
1  1 


M.  .  n.  M.  , 

(DK  1_iti)  1  2Xp[-DK  t.jJ 


M .  . 

E  [N . ]  =  DK  1_it. 
1  1 


3.  Estimation  of  Parameters: 


Likelihood  function: 


L (n^ , n2 » • • • / }  —  P  ^ { N ^  n2,**’,^m  nm  ^ 

Mi-1  ni  Mi-i 

m  (DK  1  xt.)  exp [-DK  1  At. ] 

'  tli - - 


MLE  of  parameters  D  and  K  can  be  obtained  by  solving  the 

following  pair  of  equations  simultaneously 

.  m  m  M.  , 

5  .1  "t  -  .1  *  1_1ti  -  o 

1=1  1=1 


n  m  m  M.  .-1 

V  "A-l  '  D  Jj  Mi-1K  1_  ‘i  “  0 


The  above  two  equations  yield 


y  n . 

1  m  i  - 1  1 

k  l  Vi-i  -  'tt: 
i-i  y  k  1 


i  m  M.  ,  - 1 

- )  f  M.  .K  1  t. 

m  M.  ,  l-l  i 

l  K  l"1ti  11 

i=l 


n-  23 


4W 


from  which  we  can  find  K  and  hence 


m 

I  n. 
i-1  1 


m  M.  . 

I  K 

i  =  l  1 


Performance  Measures: 

After  i-th  debugging  interval  the  failure  rate  is 


i  1 

R ( t)  =  Pr{No  failures  in  (  £  t. ,  £  t.  +  t] } 

i=  1  1  i=l  1 

M. 

-DK  1t 


MTTF  =  R(t)dt  =  — s—  =  Mean  time  to  failure  after  i-th 

M  . 


DK  1  debugging  interval. 


D-8  Modified  Schick-Wolverton  Model  (Model  FC9) 

1 .  Assumptions 

a.  Initial  fault  content: 

An  unknown  fixed  quantity  N 

b.  Independence  of  faults: 

.  Each  fault  in  the  program  is  independent  of 

other  faults  and  each  of  them  is  equally  likely 
to  cause  a  failure  during  testing. 

'lumber  of  faults  discovered  in  any  testing  interval 
is  independent  of  that  in  any  other  intervals. 

c.  Fault  removal  process: 

All  detected  faults  are  removed  with  certainty  at 
the  end  of  each  testing  interval. 

Fault  removal  tine  is  negligible 
.  No  new  faults  are  introduced  during  the  fault 
removal  process. 

d.  Hazard  function 


The  fau 

It 

occurrence  rate 

or  t 

he  hazard  function 

dur i ng 

a 

testing  interval 

i  s  p 

roportional  to  the 

number 

of 

remaining  fault 

s  at 

the  beginning  of 

this  in 

te 

rval  and  to  the 

tota  1 

time  previously 

spent  i 

n 

testing  (includi 

ng  an 

"avereged"  error 

search 

ti 

me  during  the  current 

testing  interval) 

Speci f i 

ca 

lly,  the  hazard 

funct 

ion  during  the  ith 

tes  ting 

i 

nterval  is  given 

by 
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Z(ti)  =  <fr[N- Mi_1]  [Ti_1  +  ti/2] 


where 


M.  =  ?  N.  = 


total  number  of  faults 

J  i=l  1 

removed  up  to  the  end  of 
the  j-th  interval 
t.  =  i-th  debugging  interval 
TV  _ i  =  cumulative  test  time  through  (i-l)th 


i-1 

interval  =  [  t . ;  Tn  =  0 

j  =  l  3 

<t>  =  proportionality  constant. 


2 .  Basic  Formulae 

Number  of  faults  detected  in  the  i-th  testing  inter 
val  of  length  t^  has  a  Poisson  distribution  with  rate 

<t»  [  N  —  M  J  [Ti_1  +  —  ]  i.e. 

2  t.  ni  ^  • 

{4>  tti_1  +-^]ti)  exp[-4>  [N-Mi_1)  (Ti_1  -j-)  t±  ] 

pr{Ni  =  n.}  - - - JTF — - 

and 

t . 

E[Ni]  =  0[N  -  M._1][Ti_1  +  ^]ti 

3.  Estimation  of  Parameters 

The  number  of  faults  detected  in  each  test  interval 
is  observed.  Supnose  the  number  observed  during  m  such 
intervals  t^t,,  ...,tm  are  nlf  n2,  and  nm,  respectively. 

Then  the  likelihood  of  observing  n^  faults  in  the  first  interval, 
interval,  in  the  second  and  so  on,  is  given  by 
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L ( n  ^  ,  n  2 .  nm/ ^  ^  1 7 , . . . , t^)  P r  t  N . 


"r- 


2  2  ’  m  m 


t .  n .  t . 

m  {^tN-Mi_1]  [Ti_1  +  -^J  ti}  1exp[-J>  (N-Mi_1)  (Ti_1  +  ^)  t.  ] 


i=l 


n .  ! 

l 


MLE  of  parameters  N  and  <J>  can  be  obtained  by  solving  the 
following  pair  of  equations  simultaneously 


,  m  m 

-  [  n.  -  l  (N  -  M.  J(T.  ,  +  t,/2)t.  =  0 

*  i=l  1  i=l 


i-1  '  l-l  i'  i 


and 


m  m 

s^ir-  -  «  1,  (Ti-i  *  V2,ti  -  0 
1=1  1-1  1-1 


The  above  two  equations  yield 

n  m 


m 


n  m  t . 

.1  nt(  l  (Ti  +-^)t  ) 
i=l  1  i  =  l  1  x 


i=l  N-Mi-1 


n 


—  =0 


Jj  (N-Mi-i)<Ti-i  +  V2)ti 


from  which  we  can  find  N.  Therefore 


<t>  = 


n  n . 

y  1 

. [ ,  n  -  ?r  r 

1=1  i-i 


J  <Ti-i  +t i/2>ti 


4 .  Performance  Measures: 

After  (i-l)th  debugging  interval  the  failure  rate  is 
t  [N  -  tTi_i  +  t i / 2 )  ,  therefore 
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UNCLASSIFIED 


F/G  9/2 
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NATIONAL  RURIAU  Ot  STANDARDS  I9N.' A 


i-1  i  ' 

Reliability  R(t)  *  Pr{no  failure  during  (  £  t  , 

m*l  m  _ 

.  e-^N-Mi_i]/(ti_i+ti/2)dti 

[Tj,_1  +  t  /4]t 

=  exp{-<t»[N -Mi_1]  [Ti_1 +  |]t} 

QO 

MTTF  after  (i-l)th  debugging  interval  =  J  R(t)dt. 


5 .  Data  Requirements 

Data  on  testing  interval  lengths  anci 

the  corresponding  number  of  failures  n^n,,  . . . ,nm  are 
needed  to  estimate  the  parameters  N  and  <t> . 


6 .  Model  Applicability 

This  model  could  be  used  when  the  testing  effort 
is  constant  during  a  given  testing  interval  and  if  the 
underlying  assumptions  are  satisfied. 


7 .  Relevant  References 

The  model  was  suggested  in  Lipon  [LIP74]  and  used 
to  analyze  some  failure  data  by  Sukert  [SUK76].  Also, 
the  performsnce  of  this  model  was  compared  in  ISUK76] 
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D-9 .  Generalized  Poisson  Model (Model  FC10) 

1 .  Assumptions 

a.  Initial  Fault  Content: 

An  unknown  fixed  quantity  N 

b.  Independence  of  Faults 

Each  fault  in  the  program  is  independent  of  the 
other  faults  and  each  of  them  is  equally  likely  to 
occur.  Number  of  faults  discovered  in  a  testing 
interval  is  independent  of  the  number  in  another 
interval . 

c.  Error  Removal  Process; 

.  When  faults  are  removed  they  are  removed  with 
certainty  at  the  ends  of  the  debugging  intervals. 

.  No  new  faults  are  introduced  during  the  correction 
time . 

.  Time  to  correct  faults  is  negligible. 

d.  Hazard  Function; 

The  number  of  faults  Ni  detected  in  the  ith  testing 
interval  has  a  Poisson  distribution  with  mean  value 
function 

m(til=  EtNj]  -  *[N  -  M. _ j ] t? , 

where, 

M.  =  N.  =  total  number  of  faults  removed 
J  i=l  up  to  the  end  of  the  j-th 

debugging  interval. 

t^  =  i-th  debugging  interval 

♦  =  constant  of  proportionality 

a  ■  constant 
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i.e. 

ZttL\  «  *a[N  - 


2 .  Basic  Formulae: 

Number  of  faults  detected,  N^,  in  the  i-th  debugging 

interval  t^  has  a  Poisson  distribution,  i.e. 

n. 

(4>  CN  -  M.  ,)t?}  1  exp [-$  (N  -  M.  ,)t?] 

Pr  {N .  =  n .  }  =  - h—  - 

1  i  n^ I 

and 

E TNi ]  i  m(ti)  =  4>(N-Mi-;i)t“ 

3.  Estimation  of  Parameters; 

Likelihood  Function 

L(nl'n2' * • ’ ,nm}  *  Pr*Nl  *  ni,N2  *  n2 '  **•  Nm  nm* 

n, 

m  {<fr(N  -  M.  Jtj]  1  „ 

=  n  - — l -  exp  [ — 4>  (N  -  M.  )t?] 

i_l  nj_  •  i_x 


MLE  of  parameters  N,  <J>  and  a  can  be  obtained  from  the  following 
set  of  three  equations 


m  n. 

y  1 

m 

-  *  l 

i*l 

ih  N‘*i-i 

i  ro 

m 

T  l  ni  - 
♦  i-1  1 

l  (N 
i=l 

and 


l  n.tnt.  -  ♦  ?  (N  -  M.  , ) t?£nt.  *  0 

i»l  1  1  i=l  iiii 
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D-10.  IBM  Binomial  Model  (Model  FC11) 


1 .  Assumptions 

a.  Initial  fault  content: 

The  number  of  faults  in  the  software  system 
(existing  before  the  test  phase)  is  an  unknown 
fixed  quantity  N. 

b.  Independence  of  faults: 

.  Each  failure  of  the  software  is  caused  by  one 

fault  and  each  of  them  is  equally  likely  to  cause 
a  failure  during  a  specified  unit  interval  of 
testing . 

.  Probability  of  detecting  any  one  fault  during  a 
specified  unit  interval  of  testing  is  constant 
over  all  test  occasions  and  independent  of  other 
fault  detections. 

c.  Fault  removal  process: 

.  Fault  removal  time  is  negligible 

New  faults  can  be  introduced  durinp  the  fault 
correction  process. 

Number  of  faults  introduced  on  any  test  occasion 
is  proportional  to  the  number  of  faults  detected. 

d.  Testing  Process: 

The  testing  phase  of  the  software  system  (module) 
is  divided  into  a  number  of  test  occasions. 

.  Each  test  occasion  of  the  software  system  (module) 
is  further  divided  into  a  number  of  unit  intervals. 
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Basic  Formulae 


a)  Binomial  module  level  model 

The  distribution  of  the  number  of  faults  detected 

during  the  ith  test  occasion  in  module  j  follows  a 

binomial  distribution  with  parameters  and  q^.  i.e. 

N . .  x .  .  S . . - x . • 

P("tj  "  xiJ>  -  Oij3  U  -  qjj)  11  1J 


where 


expected  number  of  faults  remaining 
in  module  j  at  the  beginning  of 
test  occasion  i. 

number  of  faults  detected  in  module 
j  during  test  occasion  i. 
fault  detection  probability  for  test 
occasion  i  of  module  j . 


j 


where  u>j  »  weight  assigned  to  module  j 
(ratio  of  the  size  of  module 
j  to  that  of  the  total  system).  If  all 
of  the  modules  of  the  system  are  under 
test  on  occasion  i,  then 

■ 1 

N  »  total  number  of  faults  in  the  system 
at  the  beginning  of  the  first  test 
occasion 
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N.  ,  .  »  cumulative  number  of  faults  detected 
1  ”  1  >  J 

in  module  j  through  test  occasion  i-1. 
a  ■  probability  of  correcting  faults  without 
introducing  additional  faults. 


where  q 


error  detection  probability  during  one  unit 

interval  of  testing 

total  number  of  unit  intervals. 


E(nij] 


b) 


Ni 


Binomial  system  level  model 

The  distribution  of  the  number  of  faults  detected 
during  the  ith  test  occasion  of  the  software  system 
follows  a  binomial  distribution  with  parameters  5L  and 
qt,  i-e. 


N.  x.  N. -x. 

p{ni -  v =  <x;>  ^l(i  -  qi> 1  1 

where  NL  =  expected  number  of  faults  remaining  in  the 
system  at  the  beginning  of  test  occasion  i. 
n^  =  number  of  faults  detected  in  the  system 


during  test  occasion  i 
fault  detection  probab: 
occasion  of  the  system. 


q^  *  fault  detection  probability  during  ith  test 


jcj. 


N-  -  »  l  (w.N  -  cxN.  ..  .) 
jej.  -1  1  1'1 


ij 
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where 


set  of  modules  tested  on  occassion  i. 


1  -  (1-q) 


*i 


where  t^ 


l  t.  .  =  system  test  effort  on  occasion  i 
jeJ.  ^ 

(total  number  of  unit  test  inter¬ 
vals  for  all  of  the  modules  tested 
on  occasion  i) 


E[n.] 


N.q. 

ini 


3 .  Estimation  of  Parameters 

a)  Binomial  module  level  model: 

Suppose  n  »n.,,  n  .  are  the  number  of  soft- 

11  “  k  3 

ware  faults  detected  in  J  modules  on  y  test  occasions. 
Then  the  likelihood  function  of  N,  q  and  a  given  the 
actual  number  of  observed  faults  n. .  is 


L(N,q  ,a|nij ;  i  =  1,2,  ...k;  j  3  1,2,  J) 


k  J  N. .  n. . 

H  H  (  lJ)q*}J(l  -  q^O 
i*l  j-1  n.j 


N.  .  -n.  - 
il  ij 


li  1 


13 


MLE  of  parameters 
solving  the  following 


N,  q  and  a  can  be  obtained  by 
three  equations  simultaneously. 


3£n  L  _ 

= 


k 

l 

ial 


J 

l 


[Wj  £n 


N.  . 

_JLL 


N- 


ij 


-  n 


w^tii«.n(l-q)  ] 


11 
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3ln  L 
3q 


k  J  n. . t.  . 

to  -  i  i  t — ii_n 

i-l  j«l 


1  -  (l-q) 


i; 


3in  L  -  n 
3a  ""  u 


k  J  N.  . 

I  l  [N.  .in  T— LL-  +  N  .t..tn(l-q)1 

i-l  j-i  1  v  .  -  n  1  if)  l) 


n  ij 


b)  Binomial  system  level  model: 

Suppose  nj,  n2,  •••,  are  the  numbers  of  software 

faults  detected  in  the  software  svstem  on  k  test  occasions 
J 

(n.  *  7  n..).  Then  the  likelihood  function  of  N,  q  and 

1  i*l  11 

a,  given  the  actual  number  o *  observed  faults  n^,  is 


L(Ntq,a|ni;  i  «  1,2,  . . . ,  k) 

k  N.  n-  N.-n. 

-  n  (  l)  q^Cl-qj) 

i-l  ni  1  1 


MLE  of  parameters  N,  q  and  a  can  be  obtained  by 
solving  the  following  three  equations  simultaneously. 

k 


3nn  L  „ 

3TT 


l  in  [( — - — )  *  t.in(l-q)]  l  w. 
i-l  Ni-ni  1  jeJi  J 


3Hn  L 


S-JL  -  l  [  — 

i-l  \  , 


Vi 


(l-a)  1 


— *  w  • 0 


32-n  L 


k  N. 

«  l  (in  - - —  ♦  t .  i  n  ( q )  ]  l  N.  ,  . 

i-l  «i  -  n.  1  jeJ.  1  L>} 


••  \5 


D-35 


4 .  Performance  Measures: 

.  Expected  number  of  software  faults  detected  during 
test  occasion  i  is  given  by 

E[n^^]  =  N^-q^  (module  level  model) 

E[r.^]  =  N^q^  (system  level  model) 


Reliability 

Suppose  that  we  wish  to  evaluate  the  reliability 
of  the  software  system  under  test  at  the  end  of  k  test¬ 
ing  occasions.  Let  t^+^  be  the  time  interval  for  the 
next  test  occasion.  Then  the  probability  that  no 
fault  will  occur  during  the  (k+l)st  test  occasion,  i.e., 
the  reliability,  is  given  by 


R(tk+i) 


-  (1  -  q) 


H+l^k+P 


5 .  Data  Requirements 

The  data  needed  are  the  number  of  faults  detected  during 
each  test  occasion  (for  the  system  level  model)  or  the  number 
of  faults  detected  during  each  test  occasion  in  each  module 
under  test  (for  the  module  level  model). 

6 .  Model  Applicability 

This  model  can  be  used  during  unit  testing,  integration 
testing  and  system  testing,  where  the  underlying  assumptions 
are  satisfied. 

7 .  Relevant  References 

The  model  and  the  detailed  data  analyses  are  given  in 


[ BRO80  ]  . 
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D-ll.  IBM  Poisson  Model  (Model  FC12) 


1 .  Assumptions 

a.  Initial  fault  content: 

.  The  number  of  faults  in  the  software  system  (existing 
before  the  test  phase)  is  an  unknown  fixed  quantity  N. 

b.  Independence  of  faults: 

.  Each  failure  of  the  software  is  caused  by  one  fault 
and  each  of  them  is  equally  likely  to  cause  a  failure 
during  a  specified  unit  interval  of  testing. 

.  Probability  (proportionality  factor  <f>)  of  detecting 
any  one  fault  during  a  soecified  unit  interval  of 
testing,  is  constant  over  all  test  occasions  and 
independent  of  other  fault  detections. 

c.  Fault  removal  process: 

Fault  removal  time  is  negligible 

New  faults  can  be  introduced  during  the  fault 

correction  process. 

.  Number  of  faults  introduced  on  any  test  occasion 
is  proportional  to  the  number  of  faults  detected. 

d.  Testing  process: 

The  testing  phase  of  the  software  system  (module) 
is  divided  into  a  number  of  test  occasions. 

.  Each  test  occasion  of  the  software  system  (module) 
is  further  divided  into  a  number  of  unit  intervals. 
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Basic  Formulae 


follows  a  Poisson  distribution  with  parameter  NL<Jk 


-N,<p. 


1  1  rC, 


P{n.  =  x,}  = 


xi 

(N.^)  1 


xi! 


where,  =  expected  number  of  faults  remaining  in 

the  system  at  the  beginning  of  test 
occasion  i. 

n^  =  number  of  faults  detected  in  the  system 
during  test  occasion  i. 

^  =  fault  detection  rate  of  each  fault  in 
the  system  during  test  occasion  i 

♦  i  -  1  -  (W)*1 

where,  t-  =  £  t..  *  system  test  effort  on  occasion  i 

jeJi  ° 

E[n.]  =  Mi 


Estimation  of  Parameters 
a)  Poisson  module  level  model: 

Suppose  n^,  n^*  ...,  n^j  are  the  number  of 
software  faults  detected  in  J  modules  on  k  test  occasions, 
Then  the  likelihood  function  of  N ,$  and  a  given  the  actual 
number  of  observed  faults  n^  is 

L(N,p,ot|n^jj  i  ~  1,2,  *  •  •  ,  k,  j  —  1,2,  ■»»  <J) 

n .  .  -N . . 0 • - 
k  J  CNiid>ii )  1Je  1J  1J 

=  Tt  TT  - ^ - * - - - 


i-1  3-1  ni j 


MLE  of  parameters  N,  4>  and  a  can  be  obtained  by 
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solving  the  following  three  equations  simultaneously. 

k  J  n. 


3£n  L  _  n 

—HT  =  0  “ 


l  l  ^  I  r1'  4>iJ 

i-1  j-1  J  Nm  11 


3£n  L 
3  0 


k  J 


=  l  l  ti.(l 

i=  1  7=1  J 


t. .  n. . 

VI  r _ LL 


-*)  ■’  I- 


-  N.  .  } 


3£n.  L  _  n 
3a 


1  -  (1-0) 
y  y  n.  ,  .  [  — u  -  *. .  i 

i-l  j-1  1'1*3  ft,. 


t  .  .  11 

1  1 


k  J 


n .  . 


17 


b)  Poisson  system  level  model: 

Suppose  n^  ,  n^ ,  .  ..,  n^.  are  the  numbers  of  soft¬ 
ware  faults  detected  in  the  software  system  on  k  test 
occasions . 

Then  the  likelihood  function  of  N,  0  and  a  given  the 
actual  number  of  observed  faults  n^  is 


L(N, 0 ,a  )  n^ ,  i  -  1,2,  ...,  k) 

n.  -N^i 
k  (Ni0i)  1  e  1  1 


i=  1 


n.  : 


MLE  of  the  parameters  N,  0  and  a  can  be  obtained 
by  solving  the  following  three  equations  simultaneously 


’if^0  -  .id  -  *i> 

^  i=  1  jcJ;  -1  N,  1 


3S.n  L  _ 
30 


k  t.  n. 

=  0=  l  t.  (1-0)  1  t - - - 7  *  Ni: 

i  =  1  1  -  d-0)  1  * 
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3  Jin 


k 

I  (  l 

i-1  jej. 


n . 


Ni-1 


.Hr1 


4 .  Performance  Measures: 

.  Expected  number  of  software  faults  detected  during 
test  occasion  i  is  given  by 


E[n. . 1  =  N.  ••<)>., 
ij  1.3  13 


Etn^J  =  Nid>± 


(module  level  model) 
(system  level  model) 


Reliability 

Consider  that  at  the  end  of  K  testing  occasions 
we  wish  to  evaluate  the  reliability  of  the  software 
system  under  test.  Let  tK+1  be  the  time  interval 
for  the  next  test  occasion,  then  the  probability 
that  no  fault  will  occur  during  (K+l)th  test  occasion 
is  given  by 


^Vi1  ■  e 


K+lyK+l 


where  <t>K+1  =  1  -  (l-<j>) 


'K+l 


5 .  Data  Requirements 

The  data  needed  are  the  number  of  faults  detected  during 
each  test  occasion  (for  the  system  level  model)  or  the 
number  of  faults  detected  during  each  test  occasion  in 
each  module  under  test  (for  the  module  level  model). 
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fco-  -  - 


6 .  Model  Applicability 

This  model  can  be  used  during  unit  testing,  integration 
testing  and  system  testing,  where  the  underlying  assumptions 
are  satisfied. 

7 „  Relevant  References 

The  model  and  the  detailed  data  analyses  are  given  in 
[ BRO80 ] . 
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APPENDIX  E 


DETAILS  OF  COMBINATORIAL  MODELS 

In  this  appendix  we  provide  details  of  the  underlying 
assumptions  and  other  relevant  concepts  for  the  fault  seeding 
and  input  domain  based  models  discussed  in  Section  5. 


E.l  Mills'  Hypergeometric  Model  (Model  FS1) 


Assumptions 

Faults  are  seeded  at  random.  Each  portion  of  the  program 
has  the  same  probability  of  having  a  seeded  fault. 

.  Probability  of  detecting  an  indigenous  fault  is  the  same 
as  that  of  detecting  a  seeded  fault. 


Basic  Formulae 
Let 


ns  *  number  of  seeded  faults 

xs  *  number  of  seeded  faults  detected  during  testing 
nj  »  total  number  of  indigenous  faults 

Xj  ■  number  of  indigenous  faults  detected  during  testing 
Then  the  joint  probability  of  finding  x^.  indigenous  faults 
and  xg  seeded  faults  in  n ^  +  ng  tests  is  given  by  the  hyper- 
r  trie  distribution  as  follows 

nI  ns 
(*)(*) 


Cj  indigenous  faults  andl  ^Xj^x 
_  seeded  faults  in  (nI  +  n 

S  v  _  ♦  X 


S 

S) 


n_+n  tests 
i  s 


E  - 1 


The  maximum  likelihood  estimate  of  is  given  by 


Lipow's  Extension  [LIP72] 

The  probability  of  finding  xT  indigeneous  faults,  x_  seeded 

X  s 

faults,  and  (therefore)  N-x^-Xg  tests  with  no  faults  found  in  N 
statistically  independent  tests  is  given  by 


PN(Xi'Xs;q,ni,ns) 


■  (xi+xs)q 


XT  +  XS  N_xrx3 

1  S(l-q)  1  a 


nI  ns 

(*) (‘ J  n  +n 
xl  xs  Iv  s 


nI+ns 
(  ) 

xI+xs 


s  N  i 

1  (?)qX(l-q) 

i  _ n  • 


i=0 


The  mle’s  of  nT  and  q  are  given  by 


XI  +  xs 


and 


xTn 
I  s 


if  Xr  +  X  >  1 

I  s  — 

if  xT  +  x  ■  0 
1  s 

if  xg  =  0. 


Basin's  Extension  [BAS74] 

Basin  suggested  a  method,  the  so-called  two-state 
edit  procedure,  where  one  programmer  searches  for  faults  and 
records  n^  faults  out  of  a  total  of  N  unknown  indigenous 
farlts.  A  second  programmer  edits  the  program  independently 
to  record  r  faults  out  of  N  (unknown)  faults.  The  two 
lists  of  faults  are  then  compared.  The  probability  of  k 
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faults  in  the  second  programmer  being  included  in  the 
first  programmer's  list  is  also  given  by  the  hypergeometric 
distribution,  i.e. 


<,k(N) 


n.  N-n, 

<k>< 


'r-k 


) 


TT 


The  mle  of  N  is  then  given  by 
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SELECTED  SOFTWARE  ENGINEERING  TERMS* 


Abort: 

To  terminate  a  process  prior  to  completion. 

Acceptance  criteria: 

The  criteria  a  software  product  must  meet  to  successfully  complete 
a  test  phase  or  meet  delivery  requirements. 

Acceptance  testing: 

Formal  testing  conducted  to  determine  whether  a  system  satisfies  its 
acceptance  criteria  and  to  enable  the  customer  to  determine  whether 
to  accept  the  system.  See  also  qualification  testing,  system  testing. 

Accuracy: 

(1)  A  quality  of  that  which  is  free  of  error.  (ISO) 

(2)  A  qualitative  assessment  of  freedom  from  error,  a  high 
assessment  corresponding  to  a  small  error.  (ISO) 

(3)  A  quantitative  measure  of  the  magnitude  of  error,  preferably 
expressed  as  a  function  of  the  relative  error,  a  high  value 
of  this  measure  corresponding  to  a  small  error.  (ISO) 

(4)  A  quantitative  assessment  of  freedom  from  error. 

Algorithm: 

(1)  A  finite  set  of  well-defined  rules  for  the  solution  of  a  problem 
in  a  finite  number  of  steps,  e.g.,  a  complete  specification  of  a 
sequence  of  arithmetic  operations  for  evaluating  sin  x  to  a 
given  precision.  (ISO) 

(2)  A  finite  set  of  well-defined  rules  which  gives  a  sequence  of 
operations  for  oerforming  a  specific  task. 

Analytical  model: 

A  representation  of  a  process  or  phenomenon  by  a  set  of  solvable 
equations.  See  also  simulation. 

Application  software: 

Software  specifically  produced  for  the  functional  use  of  a  computer 
system,  e.g.,  software  for  navigation,  gun  fire  control,  payroll, 
general  ledger,  etc.  Contrast  with  system  software. 

Assignment  statement: 

An  instruction  used  to  express  a  sequence  of  operations,  or  used  to 
assign  operands  to  specified  variables,  or  symbols,  or  both.  (ANSI) 


Taken  from  "A  Glossary  of  Software  Engineering  Terminology" (IEEE 
Project  729),  IEEE  Inc.,  New  York  10017. 
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Automated  design  tool: 

A  software  tool  which  aids  in  the  synthesis,  analysis,  modeling,  or 
documentation  of  a  software  design.  Examples  include  simulators, 
analytic  aids,  design  representation  processors,  and  documentation 
generators. 

Automated  test  case  ..enerator: 

See  automated  test  generator. 

Automated  test  data  generator: 

See  automated  test  generator. 

Automated  test  generator: 

A  Software  tool  that  accepts  as  input  a  computer  program  and  test 
criteria,  generates  test  input  data  that  meet  these  criteria,  and, 
sometimes,  determines  the  expected  results. 

Automated  verification  system: 

A  software  tool  that  accepts  as  input  a  computer  program  and  a  repre¬ 
sentation  of  its  specificati <n,  and  produces,  possibly  with  human  help, 
a  correctness  proof  or  disproof  of  the  program.  See  also  automated 
verification  tools. 

Automated  verification  tools: 

A  class  of  software  tools  used  to  evaluate  products  of  the  software 
development  process.  These  tools  aid  in  the  verification  of  such 
characteristics  as  correctness,  completeness,  consistency,  traceability , 
testability,  and  adherence  to  standards.  Examples  include  design 
analyzers,  automated  verification  systems,  static  analyzers,  dynamic 
analyzers,  and  standards  enforcers. 

Availability: 

(1)  The  probability  that  software  will  be  able  to  perform  its  desig¬ 
nated  system  function  when  required  for  use. 

(2)  The  ratio  of  system  up-time  to  the  total  operating  time. 

(3)  The  ability  of  an  item  to  oerform  its  designated  function  when 
required  for  use.  (ANSI/ASQC  A3-1978) 

Availability  model: 

A  model  used  for  predicting,  estimating,  or  assessing  availability. 
Baseline: 

(1)  A  specification  or  product  that  has  been  formally  reviewed  and 
agreed  upon,  that  thereafter  serves  as  the  basis  for  further 
development,  and  that  can  be  changed  only  through  formal  change 
control  procedures. 

(2)  A  configuration  identification  document  or  a  set  of  such  documents 
formally  designated  and  fixed  at  a  specific  time  during  a  Cl's  life 
cycle.  Baselines,  plus  approved  changes  from  those  baselines,  con¬ 
stitute  the  current  configuration  identification.  For  configuration 
management  there  are  three  baselines,  as  follows: 
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a)  Functional  baseline.  The  initial  approved  functional 
configuration. 

b)  Allocated  baseline.  The  initial  approved  allocated  configuration. 

c)  Product  baseline.  The  initial  approved  or  conditionally 
approved  product  configuration  identification.  (DoD-STD  480A) 

Bottom-up  design; 

The  design  of  a  system  starting  with  the  mo9t  basic  or  primitive 
components  and  proceeding  to  higher  level  components  that  use  the 
lower  level  ones.  Contrast  with  top-down  design. 


Bug: 

See  fault. 

Bug  seeding: 

See  fault  seeding. 

Build: 

An  operational  version  of  a  software  product  incorporating  a  specified 
subset  of  the  capabilities  that  the  final  product  will  include. 

Certification: 

(1)  A  written  guarantee  that  a  system  or  computer  program  complies 
with  its  specified  requirements. 

(2)  A  written  authorization  which  states  that  a  computer  system  is 
secure  and  is  permitted  to  operate  in  a  defined  environment 
with  or  producing  sensitive  information. 

(3)  The  formal  demonstration  of  system  acceptability  to  obtain 
authorization  for  its  operational  use. 

(4)  The  process  of  confirming  that  a  system,  software  subsystem,  or 
computer  program  is  capable  <f  satisfying  its  specified  require¬ 
ments  in  an  operational  environment.  Certification  usually  takes 
place  in  the  field  under  actual  conditions,  and  is  utilized  to 
evaluate  not  only  the  software  itself,  but  also  the  specifica¬ 
tions  to  which  the  software  was  constructed.  Certification  extends 
the  process  of  verification  and  validation  to  an  actual  or  simu¬ 
lated  operational  environment. 

(5)  The  procedure  and  action  by  a  duly  authorized  body  of  determining, 
verifying,  and  attesting  in  writing  to  the  qualifications  of 
personnel,  processes,  procedures  or  items  in  accordance  with 
applicable  requirements.  (ANSI/ASQC  A3-1978) 

Chief  programmer: 

The  leader  of  a  chief  programmer  team;  a  senior-level  programmer 
whose  responsibilities  include  producing  key  portions  of  the  soft¬ 
ware  assigned  to  the  team,  coordinating  the  activities  of  the  team, 
reviewing  the  work  of  the  other  team  members,  and  having  an  overall 
technical  understanding  of  the  software  being  developed. 
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Chief  programmer  team: 

A  software  development  group  that  consists  of  a  chief  programmer,  a 
backup  programmer,  a  secretary/librarian  and  additi  dial  programmers 
and  specialists  as  needed  and  employs  support  procedures  designed  to 
enhance  group  communication  and  make',  optimum  use  of  each  member's  skills. 

Code: 

(1)  A  set  of  unambiguous  rules  specifying  the  manner  in  which  data  may 
be  represented  in  a  discrete  form.  (ISO) 

(2)  To  represent  data  or  a  computer  program  in  a  symbolic  form  that 
can  be  accepted  by  a  processor.  (ISO) 

(3)  To  write  a  routine.  (ANSI) 

(4)  Loosely,  one  or  more  computer  programs,  or  part  of  a  computer 
program. 

(5)  The  encryption  of  data  for  security  purposes. 

Code  generator: 

A  program  or  program  function,  of  ten  part  of  a  compiler,  which  transforms 
a  computer  orogram  from  some  intermediate  level  of  representation  (often 
the  output  of  a  parser)  into  a  lower  level  representation  such  as 
assembly  code  or  machine  code. 

Comment : 

(1)  Information  embedded  within  a  computer  program,  command  language, 
or  set  of  data  which  is  intended  to  provide  clarification  to  human 
readers  and  that  does  not  effect  machine  interpretation. 

(2)  A  description,  reference,  or  explanation  added  to  or  interspersed 
among  the  statements  of  the  source  language,  that  has  no  effect 
in  the  target  language.  (ISO) 

Complexity: 

The  degree  of  complication  of  a  system  or  system  component,  determined 
by  such  factors  as  the  number  and  intricacy  of  interfaces,  the  number 
and  intricacy  of  conditional  branches,  the  degree  of  nesting,  the  types 
of  data  structures,  and  other  system  characteristics. 

Computer: 

(1)  A  functional  unit  that  can  perform  substantial  computation, 
including  numerous  arithmetic  operations,  or  logic  operations 
without  intervention  by  a  human  operator  during  a  run.  (ISO) 

(2)  A  functional  programmable  unit  that  consists  of  one  or  mere 
associated  processing  units  and  peripheral  equipment,  that  is 
controlled  by  internally  stored  programs,  and  that  can  perform 
substantial  computation,  including  numerous  arithmetic  operations 
or  logic  operations,  without  human  intervention. 

Computer  data: 

Data  available  for  communication  between  or  within  computer  equipment. 
Such  data  can  be  external  (in  computer-readable  form)  or  resident  with¬ 
in  the  computer  equipment  and  can  be  in  the  form  of  analog  or  digital 
signals. 
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Computer  program: 

A  sequence  of  instructions  suitable  for  processing  by  a  computer. 
Processing  may  include  the  use  of  an  assembler,  a  compiler,  an 
interpreter,  or  a  translator  to  prepare  the  program  for  execution 
as  well  as  to  execute  it.  (ISO)  See  also  program. 

Computer  program  abstract: 

A  brief  description  of  a  computer  program,  providing  sufficient  informa¬ 
tion  for  potential  users  to  determine  the  appropriateness  of  the 
computer  program  to  their  needs  and  resources. 

Computer  system: 

A  functional  unit,  consisting  of  one  or  more  computers  and  associated 
software,  that  uses  common  storage  for  all  or  part  of  a  program  and 
also  for  all  or  part  of  the  data  necessary  for  the  execution  of  the 
program;  executes  user-written  or  user-designated  programs;  performs 
user-designated  data  manipulation,  including  arithmetic  operations  and 
logic  operations,  and  that  can  execute  programs  that  modify  themselves 
during  their  execution.  A  computer  system  may  be  a  dtandalone  unit  or 
may  consist  of  several  interconnected  units.  Synonymous  with  ADP 
system,  computing  system.  (ISO) 

Corrective  maintenance: 

Maintenance  performed  specifically  to  overcome  existing  faults.  (ISO) 

See  also  software  maintenance. 

Correctness: 

(1)  The  extent  to  which  software  is  free  from  design  defects  and 
from  coding  defects;  i.e.,  fault  free. 

(2)  The  extent  to  which  software  meets  its  specified  requirements. 

(3)  The  extent  to  which  software  meets  user  expectations. 

Criticality: 

A  classification  of  a  software  error  or  fault  based  upon  an  evaluation 
of  the  degree  of  impact  of  that  error  or  fault  on  the  development  or 
ooeration  of  a  system.  Often  used  to  determine  whether  or  when  a  fault 
will  be  corrected. 


Data: 

A  representation  of  facts,  concepts  or  instructions  in  a  formalized 
manner  suitable  for  communication,  interpretation,  or  processing  by 
human  or  automatic  means.  (ISO)  See  also  computer  data,  error 
data,  software  experience  data,  reliability  data. 

Debugging: 

The  process  of  locating,  analyzing,  and  correcting  suspected  faults. 
Compare  with  testing. 

Debugging  model: 

See  error  model. 


Defect: 

See  fault. 


Definition  phase: 

See  requirements  phase. 

Design: 

(1)  The  process  of  defining  the  software  architecture,  components, 
modules,  interfaces,  test  approach,  and  data  for  a  software 
system  to  satisfy  sprecified  requirements. 

(2)  The  results  of  the  design  process. 

Development  methodology: 

A  systematic  approach  to  the  creation  of  software  that  defines  develop¬ 
ment  phases  and  specifies  the  activities,  products,  verification  pro¬ 
cedures,  and  completion  criteria  for  each  phase. 

Embedded  computer  system; 

A  computer  system  that  is  integral  to  a  larger  system  whose  primary 
purpose  is  not  computational,  e.g.,  a  computer  system  in  a  weapon, 
aircraft,  command  and  control,  or  rapid  transit  system. 

Embedded  software: 

Software  for  an  embedded  computer  system. 

Error: 

(1)  A  discrepancy  between  a  computed,  observed,  or  measured  value  or 
condition  and  the  true,  specified,  or  theoretically  correct  value 
or  condition.  (ANSI) 

(2)  Human  action  which  results  in  software  containing  a  fault. 

Examples  include  omission  or  misinterpretation  of  user  require¬ 
ments  in  a  software  specification,  incorrect  translation  or  omission 
of  a  requirement  in  the  design  specif ication.  This  is  not  a 
preferred  usage. 

See  also  failure,  fault. 

Error  analysis: 

(1)  The  process  of  investigating  an  observed  software  fault  with  the 
purpose  of  tracing  the  fault  to  its  source. 

(2)  The  process  of  investigating  an  observed  software  fault  to 
identify  such  information  as  the  cause  of  the  fault,  the  phase 
of  the  development  process  during  which  the  fault  was  introduced, 
methods  by  which  the  fault  could  have  been  prevented  or  detected 
earlier,  and  the  method  by  which  the  fault  was  detected. 

(3)  The  process  of  investigating  software  errors,  failures,  and 
faults  to  determine  quantitative  rates  and  trends. 

Error  category: 

One  of  a  set  of  classes  into  which  an  error,  fault,  or  failure  might 
fall.  Categories  may  be  defined  for  the  cause,  criticality,  effect, 
life  cycle  phase  when  introduced  or  detected,  or  other  characteristics 
of  the  error,  fault,  or  failure. 
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Error  data: 

A  term  commonly  (but  not  precisely)  used  to  denote  information 
describing  software  problems,  faults,  failures  and  changes,  their 
characteristics,  and  the  conditions  under  which  they  are  encountered 
or  corrected. 

Error  model: 

A  mathematical  model  used  to  predict  or  estimate  the  number  of  remaining 
faults,  reliability,  required  test  time  or  similar  characteristics  of 
a  software  system.  See  also  error  prediction. 

Error  prediction: 

A  quantitative  statement  aoout  the  expected  number  or  nature  of  soft¬ 
ware  problems,  faults,  or  failures  in  a  software  system.  See  also 
error  model. 

Error  prediction  model: 

See  error  model. 

Error  seeding: 

See  fault  seeding. 

Execution: 

The  process  of  carrying  out  an  instruction  or  the  instructions  of  a 
computer  program  by  a  computer.  (ISO) 

Execution  time: 

(1)  The  amount  of  actual  or  central  processor  time  used  in  executing 
a  program. 

(2)  The  period  of  time  during  which  a  program  is  executing. 

See  also  run  tims. 

Execution  time  theory: 

A  theory  that  uses  cumulative  execution  time  as  the  basis  for  estimating 
software  reliability. 

Executive  program: 

See  supervisory  program. 

Failure : 

(1)  The  termination  of  the  ability  of  a  functi cnal  unit  to  perform 
its  required  function.  (ISO) 

(2)  The  inability  of  a  system  or  system  component  to  perform  a  required 
function  within  specified  limits.  A  failure  may  be  produced  when 

a  fault  is  encountered. 

(3)  A  departure  of  program  operation  from  program  requirements. 

Failure  category: 

See  error  category. 

Failure  data: 

See  error  data. 
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Failure  rate: 

(1)  The  ratio  of  the  number  of  failures  to  a  given  unit  of  measure, 
e.g.,  failures  per  unit  of  time,  failures  per  number  of  transac¬ 
tions,  failures  per  number  of  computer  runs. 

(2)  In  reliability  modeling,  the  ratio  of  the  number  of  failures  of 
a  given  category  or  severity  to  a  given  period  of  time,  e.g., 
failures  per  second  of  execution  time,  failures  per  month. 

Synonymous  with  failure  ratio. 

Failure  ratio: 

See  failure  rate. 

Failure  recovery: 

The  return  of  a  system  to  a  reliable  operating  state  after  failure. 

Fault: 

(1)  An  accidental  condition  that  causes  a  functional  unit  to  fail 
to  perform  its  required  function.  (ISO) 

(2)  A  manifestation  of  an  error^  'in  software.  A  fault,  if  encountered, 
may  cause  a  failure. 

Synonymous  with  bug. 

Fault  category: 

See  error  category. 

Fault  insertion: 

See  fault  seeding. 

Fault  seeding: 

The  process  of  intentionally  adding  a  known  number  of  faults  to  those 
already  in  a  computer  program  for  the  purpose  of  estimating  the  number 
of  indigenous  faults  in  the  program. 

Synonymous  with  bug  seeding. 

Fault  tolerance: 

The  built-in  capability  of  a  system  to  provide  continued  correct  execu¬ 
tion  in  the  presence  of  a  limited  number  of  hardware  or  software  faults. 

Formal  testing: 

The  process  of  conducting  testing  activities  and  reporting  results  in 
accordance  with  an  approved  test  dan. 

Function: 

(1)  A  specific  purpose  of  an  entity  or  its  characteristic  action.  (ANSI) 

(2)  A  subprogram  that  is  invoked  during  the  evaluation  of  an  expression 
in  which  its  name  appears  and  that  returns  a  value  to  the  point  of 
invocation.  Contrast  with  subroutine. 

Hardware : 

Physical  equipment  used  in  data  processing,  as  opposed  to  computer 
programs,  procedures,  rules,  and  associated  documentation.  Contrast 
with  software.  (ISO) 


Imperfect  debugging: 

In  reliability  modeling,  the  assumption  that  attempts  to  correct  or 
remove  a  detected  fault  are  not  always  successful. 

Indigenous  fault: 

A  fault  existing  in  a  computer  program  that  has  not  been  inserted  as 
part  of  a  fault  seeding  process. 

Inspection; 

(1)  A  formal  evaluation  technique  in  which  software  requirements, 
design,  or  code  are  examined  in  detail  by  a  person  or  group 
other  than  the  author  to  detect  faults,  violations  of  development 
standards,  and  other  problems.  Contrast  with  walk-through. 

(2)  A  phase  of  quality  control  which  by  means  of  examination,  observa¬ 
tion  or  measurement  determines  the  conformance  of  materials, 
supplies,  components,  parts,  appurtenances,  systems,  processes  or 
structures  to  predetermined  quality  requirements.  (ANSI  45.2.10-1973) 

Installation  and  checkout  phase: 

The  period  of  time  in  the  software  life  cycle  during  which  a  software 
product  is  integrated  into  its  operational  environment  and  tested  in 
this  environment  to  ensure  that  it  performs  as  required. 

Integration: 

The  process  of  combining  software  elements,  hardware  elements,  or  both 
into  an  overall  system. 

Integration  testing: 

An  orderly  progression  of  testing  in  which  software  elements,  hardware 
elements,  or  both  are  combined  and  tested,  until  the  entire  system  has 
been  integrated.  See  also  system  testing. 

Interface  testing: 

Testing  conducted  to  ensure  that  program  of  system  components  pass 
information  or  control  correctly  to  one  another. 

Life  cycle: 

See  software  life  cycle. 

Main  tainabi li ty : 

(1)  The  ease  with  which  software  can  be  maintained. 

(2)  The  ease  with  which  maintenance  of  a  functional  unit  can  be  per¬ 
formed  in  accordance  with  prescribed  requirements.  (ISO) 

(3)  Ability  of  an  item  under  stated  conditions  of  use  to  be  retained 
in,  or  restored  to,  within  a  given  period  of  time,  a  specified 
state  in  which  it  can  perform  its  required  functions  when  main¬ 
tenance  is  performed  under  stated  conditions  and  while  using 
prescribed  procedures  and  resources.  (ANS1/ASQC  A3-1978) 

Maintenance: 

See  software  maintenance. 


F-9 


Maintenance  phase: 

See  operation  and  maintenance  phase. 

Maintenance  plan: 

A  document  that  identifies  the  management  and  technical  approach  that 
will  be  used  to  maintain  software  products.  Typically  included  are 
topics  such  as  tools,  resources,  facilities,  and  schedules. 

Model: 

A  representation  of  a  real  world  process,  device,  or  concept.  See 
also  analytical  model,  availability  model,  debugging  model,  error 
model,  reliability  model,  simulation,  statistical  test  model. 

Module : 

(1)  A  program  unit  that  is  discrete  and  identifiable  with  respect  to 
compiling,  combining  with  other  units,  and  loading,  e.g.,  the 
input  to,  or  output  from,  an  assembler,  compiler,  linkage  editor, 
or  executive  routine.  (ANSI) 

(.2)  A  logically  separable  part  of  a  program. 

Mutation: 

See  program  mutation. 

Object  program: 

A  fully  compiled  or  assembled  program  that  is  ready  to  be  loaded  into 
the  computer.  (ISO)  Contrast  with  source  program. 

Operating  system: 

Software  that  controls  the  execution  of  programs.  An  operating  system 
may  provide  services  such  as  resource  allocation  scheduling,  input/ 
output  control,  and  data  management.  Although  operating  systems  are 
predominantly  software,  partial  or  complete  hardware  implementations 
are  possible.  (ISO)  An  operating  system  provides  support  in  a  single 
spot  rather  than  forcing  each  program  to  be  concerned  with  controlling 
hardware.  See  also  system  software. 

Operation  and  maintenance  phase: 

The  period  of  time  in  the  software  life  cycle  during  which  a  software 
product  is  employed  in  its  operational  environment,  monitored  for 
satisfactory  performance,  and  modified  as  necessary  to  correct  problems 
or  to  respond  to  changing  requirements. 

Operational: 

Pertaining  to  the  status  given  a  software  product  once  it  has  entered 
the  operation  and  maintenance  phase. 

Operational  reliability: 

The  reliability  of  a  system  or  software  subsystem  in  its  actual  use 
environment.  Operational  reliability  may  differ  considerably  from 
reliability  in  the  specified  or  test  environment. 

Operational  testing: 

Testing  performed  by  the  end  user  on  software  in  its  normal  operating 
environment.  (DOD  usage) 
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Parameter: 

(1)  A  variable  that  is  given  a  constant  value  for  a  specified  applica¬ 
tion  and  that  may  denote  the  application.  (ISO) 

(2)  A  variable  that  is  used  to  pass  values  between  program  routines. 

See  also  actual  parameter,  formal  parameter. 

Perfective  maintenance 

Maintenance  performed  to  improve  performance,  maintainability,  or  other 
software  attributes.  See  also  adaptive  maintenance,  corrective  mainten¬ 
ance. 

Performance; 

(1)  The  ability  of  a  computer  system  of  subsystem  to  perform  its 
functions. 

(2)  A  measure  of  the  ability  of  a  computer  system  or  subsystem  to 
perform  its  functions,  e.g.,  response  time,  throughput,  number  of 
transactions.  See  also  performance  requirement. 

Performance  evaluation: 

The  technical  assessment  of  a  system  or  system  component  to  determine 
how  effectively  operating  objectives  have  been  achieved. 

Performance  requirement: 

A  requirement  that  specifies  a  performance  characteristic  that  a  system 
of  system  component  must  possess,  e.g.,  speed,  accuracy,  frequency. 

Performance  specification: 

(1)  A  specification  that  sets  forth  the  performance  requirements  for  a 
system  or  system  component. 

(2)  Synonymous  with  requirements  specification.  (U.S.  Navy  usage) 
Program: 

(1)  A  computer  program, 

(2)  A  schedule  or  plan  that  specifies  actions  to  be  taken. 

(3)  To  design,  write,  and  test  computer  programs.  (ISO) 

Program  correctness: 

See  correctness. 

Program  extension: 

An  enhancement  made  to  existing  software  to  increase  the  scope  of 
its  capabilities. 

Program  mutation: 

(1)  A  program  version  purposely  altered  from  the  intended  version  to 
evaluate  the  ability  of  program  test  cases  to  detect  the  altera¬ 
tion.  Synonymous  with  program  mutant. 

(2)  The  process  of  creating  program  mutations  in  order  to  evaluate  the 
adequacy  of  program  test  data. 
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Program  validation: 

Synonymous  with  computer  program  validation.  See  validation. 

Proof  of  correctness: 

(1)  A  formal  technique  used  to  prove  mathematically  that  a  program 
satisfies  its  specifications.  See  also  partial  correctness, 
total  correctness. 

(2)  A  program  proof  that  results  from  applying  this  technique. 
Qualification  testing: 

Formal  testing,  usually  conducted  by  the  developer  for  the  customer, 
to  demonstrate  that  the  software  meets  its  specified  requirements. 

See  also  acceptance  testing,  system  testing. 

Quality: 

(1)  The  totality  of  features  and  characteristics  of  a  product  or 
service  that  bear  on  its  ability  to  satisfy  given  needs  (ANSI. 

ASQC  A3-1978) . 

(2)  See  software  quality. 

Quality  assurance: 

A  planned  and  systematic  pattern  of  all  actions  necessary  to  provide 
adequate  confidence  that  the  item  or  product  conforms  to  established 
technical  requirements.  (IEEE  Standard  730) 

Quality  metric: 

A  quantitative  measure  of  the  degree  to  which  software  possesses  a 
given  attribute  which  affects  its  quality. 

Regression  testing: 

Selective  retesting  to  detect  faults  introduced  during  modification  of 
a  system  or  system  component  to  verify  that  modifications  have  not 
caused  unintended  adverse  effects,  or  to  verify  that  a  modified  system 
or  system  component  still  meets  its  specified  requirements. 

Reliability : 

(1)  The  ability  of  an  item  to  perform  a  required  function  under  stated 
conditions  for  a  stated  period  of  time  (ANSI/ASQC  A3-1978  and 

IEC  271-1974) 

(2)  See  software  reliability. 

Reliability,  numerical: 

The  probability  that  an  item  will  perform  a  required  function  under 
stated  conditions  for  a  stated  period  of  time.  (ANSI/ASQC  A3-1978) 

Reliability  assessment: 

The  process  of  determining  the  achieved  level  of  reliability  of  an 
existing  system  or  system  component. 

Reliability  data: 

Information  necessary  to  assess  the  reliability  of  software  at  selected 
points  in  the  software  life  cycle.  Examples  include  error  data  and  time 
data  for  reliability  models,  program  attributes  such  as  complexity,  and 
programming  characteristics  such  as  development  techniques  employed  and 
programmer  experience. 
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Reliability  evaluation: 

See  reliability  assessment 

Reliability  growth: 

The  improvement  in  software  reliability  which  results  from  correcting 

faults  in  the  software. 

Reliability  model: 

A  model  used  for  predicting,  estimating,  or  assessing  reliability. 

See  also  reliability  assessment. 

Requirement : 

(1)  A  condition  or  capability  needed  by  a  user  to  solve  a  problem 
or  achieve  an  objective, 

(2)  A  condition  or  capability  that  must  be  met  or  possessed  by  a 
system  or  system  component  to  satisfy  a  contract,  standard, 
specification,  or  other  formally  imposed  document.  The  set  of 
all  requirements  forms  the  basis  for  subsequent  development  of 
the  system  or  system  component.  See  also  requirements  analysis, 
requirements  pahse,  requirements  specification. 

Retirement  phase: 

The  period  of  time  in  the  software  life  cycle  during  which  support 

for  a  software  product  is  terminated. 

Robustness: 

The  extent  to  which  software  can  continue  to  operate  correctly  despite 

the  introduction  of  invalid  Inputs. 

Run  time: 

(1)  A  measure  of  the  time  expended  to  execute  a  program.  While  it 
ordinarily  reflects  the  expended  central  processor  time,  it  may 
also  include  peripheral  processing  and  peripheral  accessing 
time,  e.g.,  a  run  time  of  5  hours. 

(2)  The  instant  at  which  a  program  begins  to  execute. 

See  also  execution  time. 

Seeding: 

See  fault  seeding. 

Severity: 

See  criticality. 

Software: 

(1)  Computer  programs,  procedures,  rules,  and  possibly  associated 
documentation  and  data  pertaining  to  the  operation  of  a  computer 
system.  See  also  application  software,  system  software.  Contrast 
with  hardware. 

(2)  Programs,  procedures,  rules,  and  any  associated  documentation 
pertaining  to  the  operation  of  a  computer  system.  (ISO) 
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Software  data  base: 

A  centralized  file  of  data  definitions  and  present  values  for  data 
common  to,  and  located  Internal  to,  an  operational  software  system. 

Software  development  cycle: 

(1)  The  period  of  time  that  begins  with  the  decision  to  develop  a 
software  product  and  ends  when  the  product  is  delivered.  This 
cycle  typically  includes  a  requirements  phase,  design  phase, 
implementation  phase,  test  pahse,  and  sometimes,  installation 
and  checkout  phase.  Contrast  with  software  life  cycle. 

(2)  The  period  of  time  that  begins  with  the  decision  to  develop  a 
software  product  and  ends  when  the  product  is  no  longer  being 
enhanced  by  the  developer. 

(3)  Sometimes  used  as  a  synonym  for  software  life  cycle. 

Software  development  process: 

The  process  by  which  user  needs  are  translated  into  software  require¬ 
ments,  software  requirements  are  transformed  into  design,  the  design 
is  implemented  in  code,  and  the  code  is  tested,  documented,  and 
certified  for  operational  use. 

Software  documentation: 

Technical  data  or  information,  including  computer  listings  and  printouts, 
in  human-readable  form,  that  describe  or  specify  the  design  or  details, 
explain  the  capabilities,  or  provide  operating  instructions  for  using 
the  software  to  obtain  desired  results  from  a  software  system.  See 
also  documentation,  system  documentation,  user  documentation. 

Software  engineering: 

The  systematic  approach  to  the  development,  operation,  maintenance  and 
retirement  of  software. 

Software  experience  data: 

Data  relating  to  the  development  or  use  of  software  that  could  be 
useful  in  developing  models,  reliability  predictions,  or  other 
quantitative  descriptions  of  software. 

Software  life  cycle: 

The  period  of  time  that  starts  when  a  software  product  is  conceived 
and  ends  when  the  product  is  no  longer  available  for  use.  The  soft¬ 
ware  life  cycle  typically  includes  a  requirements  phase,  design  phase, 
implementation  phase,  test  phase,  installation  and  checkout  phase, 
operation  and  maintenance  phase,  and  sometimes,  retirement  phase. 

Contrast  with  software  development  cycle. 

Software  maintenance: 

(1)  Modification  of  a  software  product  after  delivery  to  correct  faults. 

(2)  Modification  of  a  software  product  after  delivery  to  correct  faults, 
to  improve  performance  or  other  attributes,  or  to  adapt  the  product 
to  a  changed  environment.  See  also  adaptive  maintenance,  correc¬ 
tive  maintenance,  perfective  maintenance. 
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Software  product: 

A  software  entity  designated  for  delivery  to  a  user. 

Software  quality: 

(1)  The  totality  of  features  and  characteristics  of  a  software 
product  that  bears  on  its  ability  to  satisfy  given  needs;  e.g., 
conform  to  specif ications. 

(2)  The  degree  to  which  software  possesses  a  desired  combination 
of  attributes. 

(3)  The  degree  to  which  a  customer  or  user  perceives  that  software 
meets  his  or  her  composite  expectations. 

(4)  The  composite  characteristics  of  software  that  determine  the 
degree  to  ihich  the  software  in  use  will  meet  the  expectations 
of  the  customer. 

Software  quality  assurance: 

See  quality  assurance. 

Software  reliability: 

(1)  The  probability  that  software  will  not  cause  the  failure  of  a 
system  for  a  specified  time  under  specified  conditions.  The 
probability  is  a  function  of  the  inputs  to  and  use  of  the 
system  as  well  as  a  function  of  the  existence  of  faults  in  the 
software.  The  inputs  to  the  system  determine  whether  existing 
faults,  if  any,  are  encountered. 

(2)  The  ability  of  a  program  to  perform  a  required  function  under 
stated  conditions  for  a  stated  period  of  time. 

Software  tool: 

A  computer  program  used  to  help  develop,  test,  analyze,  or  maintain 
another  computer  program  or  its  documentation,  e.g.,  automated  design 
tool,  compiler,  test  tool,  maintenance  tool. 

Source  program: 

(1)  A  computer  program  that  must  be  compiled,  assembled,  or  interpreted 
before  being  executed  by  a  computer, 

(2)  A  computer  program  expressed  in  a  source  language.  Contrast 
with  object  program.  (ISO) 

Specification  verification: 

See  verification. 

Statistical  test  model: 

A  model  that  relates  program  faults  to  the  input  data  set  (or  sets) 
which  cause  them  to  be  encountered.  The  model  also  gives  the  proba¬ 
bility  that  these  faults  will  cause  the  program  to  fail. 

Structured  design: 

A  disciplined  approach  to  software  design  which  adheres  to  a  specified 
set  of  rules  based  on  principles  such  as  top-down  design,  stepwise 
refinement,  and  data  flow  analysis. 
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Structured  program: 

A  program  constructed  of  a  basic  set  of  control  structures,  each 
one  having  one  entry  point  and  one  exit.  The  set  of  control  struc¬ 
tures  typically  includes:  sequence  of  two  or  more  instructions, 
conditional  selection  of  one  of  two  or  more  instructions  or  sequences 
of  instructions,  and  repetition  of  an  instruction  or  a  sequence  of 
instructions. 

Subprogram: 

A  program  unit  that  may  be  invoked  by  one  or  more  other  program  units. 
Examples  are  procedure,  function,  subroutine. 

Symbolic  execution: 

A  verification  technique  in  which  program  execution  is  simulated  using 
symbols  rather  than  actual  values  for  input  data,  and  program  outputs 
are  expressed  as  logical  or  mathematical  expressions  involving  these 
symbols . 


System  reliability: 

The  probability  that  a  system,  including  all  hardware  and  software 
subsystems,  will  perform  a  required  task  or  mission  for  a  specified 
time  in  a  specified  environment.  See  also  operational  reliability, 
software  reliability. 

System  software: 

Software  designed  for  a  specific  computer  system  or  family  of  computer 
systems  to  facilitate  the  operation  and  maintenance  of  the  computer 
system  and  associated  programs.,  e.g.  operating  systems,  compilers, 
utilities.  Contrast  with  application  software. 

System  testing: 

The  process  of  testing  an  integrated  hardware  and  software  system  to 
verify  that  the  system  meets  it  specified  requirements.  See  also 
acceptance  testing,  qualification  testing. 

Test  case: 

A  specific  set  of  test  data  and  associated  procedures  developed  for 
a  particular  objective,  such  as  to  exercise  a  particular  program  path 
or  to  verify  compliance  with  a  specific  requirement.  See  also  testing. 

Test  driver: 

A  driver  that  invokes  the  item  under  test  and  may  provide  test  inputs 
and  report  test  results. 

Test  log: 

A  chronological  record  of  all  relevant  details  of  a  testing  activity. 

Test  phase: 

The  period  of  time  in  the  software  life  cycle  during  which  the  components 
of  a  software  product  are  evaluated  and  integrated,  and  the  software 
product  is  evaluated  to  determine  whether  or  not  requirements  have  been 
satisfied. 


Test  plan: 

A  document  prescribing  the  approach  to  be  taken  for  intended  testing 
activities.  The  plan  typically  identifies  the  items  to  be  tested, 
the  testing  to  be  performed,  test  schedules,  personnel  requirements, 
reporting  requirements,  evaluation  criteria,  and  any  risks  requiring 
contingency  planning. 

Test  procedure: 

Detailed  instructions  for  the  set  up,  operation,  and  evaluation  of 
results  for  a  given  test.  A  set  of  associated  procedures  is  often 
combined  to  form  a  test  procedures  document. 

Test  validity: 

The  degree  to  which  a  test  accomplishes  its  specified  goal. 

Testability: 

(1)  The  extent  to  which  software  facilitates  both  the  establishment  of 
test  criteria  and  the  evaluation  of  software  with  respect  to  those 
criteria. 

(2)  The  extent  to  which  the  definition  of  requirements  facilitates 
analysis  of  the  requirements  to  establish  test  criteria. 

Testing: 

The  process  of  exercising  or  evaluating  a  system  or  system  component 
by  manual  or  automated  means  to  verify  that  it  satisfies  specified 
reauirements  or  to  identify  differences  between  expected  and  actual 
results.  Compare  with  debugging. 

Tolerance: 

The  ability  of  a  system  to  provide  continuity  of  operation  under 
various  abnormal  conditions. 

Total  correctness: 

In  proof  of  correctness,  a  designation  indicating  that  a  program's 
output  assertions  follow  logically  from  its  input  assertions  and 
processing  steps,  and  that,  in  addition,  the  program  terminates 
under  all  specified  input  conditions.  Contrast  with  partial  correct¬ 
ness. 

Utility  software: 

Computer  programs  or  routines  designed  to  perform  some  general  support 
function  required  by  other  application  software,  by  the  operating 
system,  or  by  system  users. 

Validation: 

The  process  of  evaluating  software  at  the  end  of  the  software  develop¬ 
ment  process  to  ensure  compliance  with  software  requirements.  See 
also  verification. 
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Verification: 

(1)  The  process  of  determining  whether  the  products  of  a  given  phase  of 
the  software  development  cycle  fulfill  the  requirements  established 
during  the  previous  phase.  See  also  validation. 

(2)  Formal  proof  of  program  correctness.  See  proof  of  correctness. 

(3)  The  act  of  reviewing,  inspecting,  testing,  checking,  auditing,  or 
otherwise  establishing,  and  documenting  whether  items,  processes, 
services,  or  documents  conform  to  specified  requirements.  (ANSI/ 
ASQC  A3- 1978). 

Walk-through: 

A  review  process  in  which  a  designer  or  programmer  leads  one  or  more 
other  members  of  the  development  team  through  a  segment  of  desig.n  or 
code  that  he  or  she  has  written,  while  the  other  members  ask  questions 
and  make  comments  about  technique,  style,  possible  errors,  violation 
of  development  standards,  and  other  problems.  Contrast  with  inspection. 
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MISSION 

of 

Rome  Air  Development  Center 

KADC  plant  and  execute a  research,  development,  tut  and 
selected  acquisition  programs  in  support  oh  Command,  Contort 
Communication*  and  Intelligence  (C3I)  activities .  Technical 
and  engineering  support  within  areas  oh  technical  competence 
is  provided  to  ESP  Program  Office*  IPOs)  and  other  ESP 
elements.  The  principal  technical  mission  areas  are 
communications,  electromagnetic  guidance  and  control,  sur¬ 
veillance  oh  ground  and  aerospace  objects,  intelligence  data 
collection  and  handling,  inhormation  system  technology, 
ionospheric  propagation,  solid  state  sciences,  microwave 
pkysict  and  electronic  reliability,  maintainability  and 
emyntibiJUty. 


END 

DATE 

FILMED 


DTIC 


